2021-02-08T07:05:10.457 INFO:root:teuthology version: 1.1.0-17c09965 2021-02-08T07:05:10.458 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864577 --name yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi --owner scheduled_yuriw@teuthology --verbose --description rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} -- /home/teuthworker/archive/yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864577/orig.config.yaml 2021-02-08T07:05:10.505 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-08T07:05:10.563 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864577 branch: wip-yuri3-testing-2021-02-03-1046-octopus description: rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: ceph-qa@ceph.io first_in_suite: false job_id: '5864577' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-yuri3-testing-2021-02-03-1046-octopus ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 133532e25eccb46507b425783df68c050462a609 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 133532e25eccb46507b425783df68c050462a609 rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-yuri3-testing-2021-02-03-1046-octopus sha1: 133532e25eccb46507b425783df68c050462a609 owner: scheduled_yuriw@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 133532e25eccb46507b425783df68c050462a609 sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuri3-testing-2021-02-03-1046-octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_133532e25eccb46507b425783df68c050462a609/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 133532e25eccb46507b425783df68c050462a609 targets: smithi017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqsworecZ+NXvz4RgtnimxK4DRM14Ko8sQhTWWw6hOfRi9MqTDMncDlRIIOghbLxvFInRuOuHi/3+gpd77/1ytqJx7zC+eCamEq0Mc3j1fBxDInmIyIuctj/+rpolBZnsyXtPREEPTLrXPMufVqWHV8bxtYuVelT+PKbvTLdKuawY4l1oxsDSr5aJ4aHBA1FnAe15iFNNbQzsbbi0tObW1ag3znNRh5+XUIFwuJZJbhMe4DUFSdE2/Xc+2EQ6MmlvvylPN9vjVJ1hu0aihPEPK0HJbpbk/u3aC/hlKu48Z7mokBux4Nmt9U/Jr0p02cENBoxmE//NCAZApA8SHYCil smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDT7W0N7K3cyUIC6+xOIqn6tGKNOr8qH8UlFgikfk6WJjcL5Dpz/7AZUsa109iARI6teTVv/APcQ0Q/Cy/eXRmD9EfjCc/7i52q0S0muTTdQtLeCwn6F7C47x0KZobHgG/LWZY99i/OsL29ycoraHNTeR8DOM2hS47MxIbkQVykN4+EjG1GNIltOL5s8yfy6M2L+WMALpBRJhL5H+7cU9B7Sn+SG5IJ/bqzbtsonVr2CCe0QRBHAEtbxw+kNVw8MEdP01VR9QB05RV059xowBtLhUUWFJJ5F9sy5T072VYD+x5AKKdt51TagnD0mjq3kffa6AOURoQo4fBI7l9Q2nRz tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master teuthology_sha1: 17c099650191e7aa47a7d9f502facca74f9b4d44 timestamp: 2021-02-07_16:16:23 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.29888 2021-02-08T07:05:10.563 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_133532e25eccb46507b425783df68c050462a609/qa; will attempt to use it 2021-02-08T07:05:10.564 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_133532e25eccb46507b425783df68c050462a609/qa/tasks 2021-02-08T07:05:10.564 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-08T07:05:10.565 INFO:teuthology.task.internal:Checking packages... 2021-02-08T07:05:10.586 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '133532e25eccb46507b425783df68c050462a609' 2021-02-08T07:05:10.586 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-08T07:05:10.586 INFO:teuthology.packaging:ref: None 2021-02-08T07:05:10.587 INFO:teuthology.packaging:tag: None 2021-02-08T07:05:10.587 INFO:teuthology.packaging:branch: wip-yuri3-testing-2021-02-03-1046-octopus 2021-02-08T07:05:10.587 INFO:teuthology.packaging:sha1: 133532e25eccb46507b425783df68c050462a609 2021-02-08T07:05:10.587 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-yuri3-testing-2021-02-03-1046-octopus 2021-02-08T07:05:10.763 INFO:teuthology.task.internal:Found packages for ceph version 15.2.8-294-g133532e2-1bionic 2021-02-08T07:05:10.763 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-08T07:05:10.771 INFO:teuthology.task.internal:no buildpackages task found 2021-02-08T07:05:10.771 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-08T07:05:10.783 INFO:teuthology.task.internal:Saving configuration 2021-02-08T07:05:10.799 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-08T07:05:10.809 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-08T07:05:10.826 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-08 06:59:27.775336', 'description': '/home/teuthworker/archive/yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864577', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF5kBBiYo/oLx9yUJeYihEU+xMdy5Jpy9w93DUxcgO7T', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi017.front.sepia.ceph.com'} 2021-02-08T07:05:10.843 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-08 06:59:27.775365', 'description': '/home/teuthworker/archive/yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864577', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH09k94AS1oCe/3NwY0moY/qZu2zakvjOAbE7sS1wZnJ', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi110.front.sepia.ceph.com'} 2021-02-08T07:05:10.843 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-08T07:05:10.901 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-02-08T07:05:10.901 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-02-08T07:05:10.901 INFO:teuthology.run_tasks:Running task console_log... 2021-02-08T07:05:11.023 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4095e03f28>, signals=[15]) 2021-02-08T07:05:11.023 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-08T07:05:11.031 INFO:teuthology.task.internal:Opening connections... 2021-02-08T07:05:11.031 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2021-02-08T07:05:11.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-08T07:05:11.150 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2021-02-08T07:05:11.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-08T07:05:11.225 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-08T07:05:11.233 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2021-02-08T07:05:11.306 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2021-02-08T07:05:11.306 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2021-02-08T07:05:11.353 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="Ubuntu" 2021-02-08T07:05:11.353 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-08T07:05:11.353 INFO:teuthology.orchestra.run.smithi017.stdout:ID=ubuntu 2021-02-08T07:05:11.353 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE=debian 2021-02-08T07:05:11.354 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-08T07:05:11.354 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="18.04" 2021-02-08T07:05:11.354 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-08T07:05:11.354 INFO:teuthology.orchestra.run.smithi017.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-08T07:05:11.354 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-08T07:05:11.354 INFO:teuthology.orchestra.run.smithi017.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-08T07:05:11.354 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_CODENAME=bionic 2021-02-08T07:05:11.354 INFO:teuthology.orchestra.run.smithi017.stdout:UBUNTU_CODENAME=bionic 2021-02-08T07:05:11.355 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2021-02-08T07:05:11.371 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2021-02-08T07:05:11.422 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2021-02-08T07:05:11.422 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2021-02-08T07:05:11.470 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu" 2021-02-08T07:05:11.470 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-08T07:05:11.470 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu 2021-02-08T07:05:11.470 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian 2021-02-08T07:05:11.470 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-08T07:05:11.470 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="18.04" 2021-02-08T07:05:11.471 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-08T07:05:11.471 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-08T07:05:11.471 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-08T07:05:11.471 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-08T07:05:11.471 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=bionic 2021-02-08T07:05:11.471 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=bionic 2021-02-08T07:05:11.471 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2021-02-08T07:05:11.485 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-08T07:05:11.498 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-08T07:05:11.510 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-08T07:05:11.510 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2021-02-08T07:05:11.513 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2021-02-08T07:05:11.518 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-08T07:05:11.528 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-08T07:05:11.528 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2021-02-08T07:05:11.559 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2021-02-08T07:05:11.588 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-08T07:05:11.643 INFO:teuthology.run_tasks:Running task kernel... 2021-02-08T07:05:11.660 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-02-08T07:05:11.660 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-08T07:05:11.661 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2021-02-08T07:05:11.667 INFO:teuthology.orchestra.run.smithi017.stdout:4.15.0-134-generic 2021-02-08T07:05:11.667 INFO:teuthology.task.kernel:Running kernel on smithi017: 4.15.0-134-generic 2021-02-08T07:05:11.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get clean 2021-02-08T07:05:11.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get update 2021-02-08T07:05:12.045 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-08T07:05:12.045 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-08T07:05:12.100 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-08T07:05:12.374 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-08T07:05:12.458 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,537 kB] 2021-02-08T07:05:12.594 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,214 kB] 2021-02-08T07:05:12.894 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [910 kB] 2021-02-08T07:05:12.928 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [297 kB] 2021-02-08T07:05:12.939 INFO:teuthology.orchestra.run.smithi017.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-08T07:05:12.942 INFO:teuthology.orchestra.run.smithi017.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,877 kB] 2021-02-08T07:05:12.948 INFO:teuthology.orchestra.run.smithi017.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-08T07:05:12.949 INFO:teuthology.orchestra.run.smithi017.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,105 kB] 2021-02-08T07:05:13.010 INFO:teuthology.orchestra.run.smithi017.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [974 kB] 2021-02-08T07:05:13.036 INFO:teuthology.orchestra.run.smithi017.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [247 kB] 2021-02-08T07:05:13.045 INFO:teuthology.orchestra.run.smithi017.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [389 kB] 2021-02-08T07:05:13.062 INFO:teuthology.orchestra.run.smithi017.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-08T07:05:13.073 INFO:teuthology.orchestra.run.smithi017.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-08T07:05:13.074 INFO:teuthology.orchestra.run.smithi017.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-08T07:05:13.113 INFO:teuthology.orchestra.run.smithi017.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,714 kB] 2021-02-08T07:05:13.170 INFO:teuthology.orchestra.run.smithi017.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [362 kB] 2021-02-08T07:05:13.176 INFO:teuthology.orchestra.run.smithi017.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-08T07:05:13.176 INFO:teuthology.orchestra.run.smithi017.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-08T07:05:13.176 INFO:teuthology.orchestra.run.smithi017.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-08T07:05:14.668 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 13.0 MB in 3s (4,882 kB/s) 2021-02-08T07:05:16.176 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2021-02-08T07:05:16.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-08T07:05:16.320 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2021-02-08T07:05:16.646 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2021-02-08T07:05:16.646 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2021-02-08T07:05:16.737 INFO:teuthology.orchestra.run.smithi017.stderr:E: Unable to locate package linux-image-current-generic 2021-02-08T07:05:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-08T07:05:16.739 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-08T07:05:16.834 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2021-02-08T07:05:17.098 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2021-02-08T07:05:17.098 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2021-02-08T07:05:17.288 INFO:teuthology.orchestra.run.smithi017.stdout:The following additional packages will be installed: 2021-02-08T07:05:17.288 INFO:teuthology.orchestra.run.smithi017.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-08T07:05:17.288 INFO:teuthology.orchestra.run.smithi017.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-08T07:05:17.288 INFO:teuthology.orchestra.run.smithi017.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-08T07:05:17.289 INFO:teuthology.orchestra.run.smithi017.stdout:Suggested packages: 2021-02-08T07:05:17.289 INFO:teuthology.orchestra.run.smithi017.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-08T07:05:17.289 INFO:teuthology.orchestra.run.smithi017.stdout:Recommended packages: 2021-02-08T07:05:17.289 INFO:teuthology.orchestra.run.smithi017.stdout: thermald 2021-02-08T07:05:17.322 INFO:teuthology.orchestra.run.smithi017.stdout:The following NEW packages will be installed: 2021-02-08T07:05:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-08T07:05:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-08T07:05:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-08T07:05:17.324 INFO:teuthology.orchestra.run.smithi017.stdout:The following packages will be upgraded: 2021-02-08T07:05:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-08T07:05:17.504 INFO:teuthology.orchestra.run.smithi017.stdout:3 upgraded, 5 newly installed, 0 to remove and 11 not upgraded. 2021-02-08T07:05:17.504 INFO:teuthology.orchestra.run.smithi017.stdout:Need to get 66.0 MB of archives. 2021-02-08T07:05:17.504 INFO:teuthology.orchestra.run.smithi017.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-08T07:05:17.505 INFO:teuthology.orchestra.run.smithi017.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-08T07:05:18.497 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-08T07:05:18.738 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-08T07:05:19.695 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-08T07:05:19.695 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-08T07:05:19.696 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-08T07:05:19.990 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-08T07:05:20.014 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-08T07:05:20.536 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 66.0 MB in 3s (24.7 MB/s) 2021-02-08T07:05:20.728 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-08T07:05:20.766 INFO:teuthology.orchestra.run.smithi017.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-08T07:05:20.769 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-08T07:05:20.816 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:05:22.458 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-08T07:05:22.469 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-08T07:05:22.575 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:05:23.061 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-08T07:05:23.072 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-08T07:05:23.110 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:05:27.256 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-08T07:05:27.377 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-08T07:05:27.692 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-08T07:05:27.814 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-08T07:05:28.167 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-08T07:05:28.175 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-08T07:05:28.208 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-08T07:05:33.130 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-08T07:05:33.150 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-08T07:05:33.188 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:05:35.931 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-08T07:05:36.053 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-08T07:05:36.386 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:05:36.925 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-08T07:05:37.051 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:05:37.503 INFO:teuthology.orchestra.run.smithi017.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-08T07:05:37.503 INFO:teuthology.orchestra.run.smithi017.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-08T07:05:37.636 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:05:38.179 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:05:38.313 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-08T07:05:38.439 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-08T07:05:38.557 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-08T07:05:38.658 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:05:38.843 INFO:teuthology.orchestra.run.smithi017.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-08T07:05:38.843 INFO:teuthology.orchestra.run.smithi017.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-08T07:05:54.437 INFO:teuthology.orchestra.run.smithi017.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-08T07:05:54.437 INFO:teuthology.orchestra.run.smithi017.stdout:Sourcing file `/etc/default/grub' 2021-02-08T07:05:54.483 INFO:teuthology.orchestra.run.smithi017.stdout:Generating grub configuration file ... 2021-02-08T07:05:54.827 INFO:teuthology.orchestra.run.smithi017.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-08T07:05:54.858 INFO:teuthology.orchestra.run.smithi017.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-08T07:05:55.162 INFO:teuthology.orchestra.run.smithi017.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-08T07:05:55.201 INFO:teuthology.orchestra.run.smithi017.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-08T07:05:56.748 INFO:teuthology.orchestra.run.smithi017.stdout:done 2021-02-08T07:05:57.131 DEBUG:teuthology.orchestra.run.smithi017:> dpkg -s linux-image-generic 2021-02-08T07:05:57.197 INFO:teuthology.orchestra.run.smithi017.stdout:Package: linux-image-generic 2021-02-08T07:05:57.197 INFO:teuthology.orchestra.run.smithi017.stdout:Status: install ok installed 2021-02-08T07:05:57.198 INFO:teuthology.orchestra.run.smithi017.stdout:Priority: optional 2021-02-08T07:05:57.198 INFO:teuthology.orchestra.run.smithi017.stdout:Section: kernel 2021-02-08T07:05:57.198 INFO:teuthology.orchestra.run.smithi017.stdout:Installed-Size: 17 2021-02-08T07:05:57.198 INFO:teuthology.orchestra.run.smithi017.stdout:Maintainer: Ubuntu Kernel Team 2021-02-08T07:05:57.198 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: amd64 2021-02-08T07:05:57.199 INFO:teuthology.orchestra.run.smithi017.stdout:Source: linux-meta 2021-02-08T07:05:57.199 INFO:teuthology.orchestra.run.smithi017.stdout:Version: 4.15.0.135.122 2021-02-08T07:05:57.199 INFO:teuthology.orchestra.run.smithi017.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-08T07:05:57.199 INFO:teuthology.orchestra.run.smithi017.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-08T07:05:57.199 INFO:teuthology.orchestra.run.smithi017.stdout:Recommends: thermald 2021-02-08T07:05:57.200 INFO:teuthology.orchestra.run.smithi017.stdout:Description: Generic Linux kernel image 2021-02-08T07:05:57.200 INFO:teuthology.orchestra.run.smithi017.stdout: This package will always depend on the latest generic kernel image 2021-02-08T07:05:57.200 INFO:teuthology.orchestra.run.smithi017.stdout: available. 2021-02-08T07:05:57.200 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-08T07:05:57.201 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2021-02-08T07:05:57.208 INFO:teuthology.orchestra.run.smithi110.stdout:4.15.0-134-generic 2021-02-08T07:05:57.209 INFO:teuthology.task.kernel:Running kernel on smithi110: 4.15.0-134-generic 2021-02-08T07:05:57.209 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2021-02-08T07:05:57.411 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2021-02-08T07:05:57.656 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-08T07:05:57.688 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-08T07:05:57.738 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-08T07:05:58.008 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-08T07:05:58.106 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,537 kB] 2021-02-08T07:05:58.236 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,877 kB] 2021-02-08T07:05:58.543 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [910 kB] 2021-02-08T07:05:58.574 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [297 kB] 2021-02-08T07:05:58.584 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-08T07:05:58.592 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-08T07:05:58.593 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,105 kB] 2021-02-08T07:05:58.625 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,214 kB] 2021-02-08T07:05:58.653 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [974 kB] 2021-02-08T07:05:58.660 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [389 kB] 2021-02-08T07:05:58.672 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-08T07:05:58.679 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-08T07:05:58.679 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-08T07:05:58.683 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [247 kB] 2021-02-08T07:05:58.746 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,714 kB] 2021-02-08T07:05:58.800 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [362 kB] 2021-02-08T07:05:58.803 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-08T07:05:58.803 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-08T07:05:58.808 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-08T07:06:00.400 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 13.0 MB in 3s (4,966 kB/s) 2021-02-08T07:06:01.862 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2021-02-08T07:06:01.905 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-08T07:06:01.982 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2021-02-08T07:06:02.212 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2021-02-08T07:06:02.212 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2021-02-08T07:06:02.292 INFO:teuthology.orchestra.run.smithi110.stderr:E: Unable to locate package linux-image-current-generic 2021-02-08T07:06:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-08T07:06:02.294 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-08T07:06:02.374 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2021-02-08T07:06:02.613 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2021-02-08T07:06:02.613 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2021-02-08T07:06:02.789 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2021-02-08T07:06:02.789 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-08T07:06:02.789 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-08T07:06:02.790 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-08T07:06:02.792 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2021-02-08T07:06:02.792 INFO:teuthology.orchestra.run.smithi110.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-08T07:06:02.792 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2021-02-08T07:06:02.792 INFO:teuthology.orchestra.run.smithi110.stdout: thermald 2021-02-08T07:06:02.826 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2021-02-08T07:06:02.826 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-08T07:06:02.826 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-08T07:06:02.826 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-08T07:06:02.828 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2021-02-08T07:06:02.828 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-08T07:06:03.006 INFO:teuthology.orchestra.run.smithi110.stdout:3 upgraded, 5 newly installed, 0 to remove and 11 not upgraded. 2021-02-08T07:06:03.006 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 66.0 MB of archives. 2021-02-08T07:06:03.006 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-08T07:06:03.006 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-08T07:06:04.212 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-08T07:06:04.580 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-08T07:06:05.916 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-08T07:06:05.916 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-08T07:06:05.916 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-08T07:06:06.296 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-08T07:06:06.310 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-08T07:06:06.795 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 66.0 MB in 3s (19.0 MB/s) 2021-02-08T07:06:06.962 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-08T07:06:07.000 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-08T07:06:07.003 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-08T07:06:07.050 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:06:08.601 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-08T07:06:08.613 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-08T07:06:08.697 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:06:09.204 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-08T07:06:09.215 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-08T07:06:09.254 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:06:13.003 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-08T07:06:13.104 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-08T07:06:13.429 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-08T07:06:13.558 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-08T07:06:13.894 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-08T07:06:13.907 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-08T07:06:13.952 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-08T07:06:18.745 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-08T07:06:18.768 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-08T07:06:18.811 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:06:21.230 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-08T07:06:21.361 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-08T07:06:21.677 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:06:22.216 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-08T07:06:22.342 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:06:22.773 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-08T07:06:22.773 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-08T07:06:22.902 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:06:23.354 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:06:23.596 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-08T07:06:24.139 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-08T07:06:24.265 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-08T07:06:24.391 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-08T07:06:24.596 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-08T07:06:24.596 INFO:teuthology.orchestra.run.smithi110.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-08T07:06:39.119 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-08T07:06:39.120 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub' 2021-02-08T07:06:39.149 INFO:teuthology.orchestra.run.smithi110.stdout:Generating grub configuration file ... 2021-02-08T07:06:39.475 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-08T07:06:39.509 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-08T07:06:39.826 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-08T07:06:39.856 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-08T07:06:41.145 INFO:teuthology.orchestra.run.smithi110.stdout:done 2021-02-08T07:06:41.568 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2021-02-08T07:06:41.589 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2021-02-08T07:06:41.589 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2021-02-08T07:06:41.589 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 17 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 4.15.0.135.122 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2021-02-08T07:06:41.590 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2021-02-08T07:06:41.591 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-08T07:06:41.591 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-02-08T07:06:41.591 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-08T07:06:41.591 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-08T07:06:41.591 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2021-02-08T07:06:41.591 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2021-02-08T07:06:41.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get clean 2021-02-08T07:06:41.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get update 2021-02-08T07:06:41.896 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-08T07:06:41.903 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-08T07:06:41.963 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-08T07:06:42.041 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-08T07:06:43.882 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2021-02-08T07:06:43.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-08T07:06:44.011 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2021-02-08T07:06:44.260 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2021-02-08T07:06:44.260 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2021-02-08T07:06:44.346 INFO:teuthology.orchestra.run.smithi017.stderr:E: Unable to locate package linux-image-current-generic 2021-02-08T07:06:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-08T07:06:44.347 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-08T07:06:44.430 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2021-02-08T07:06:44.703 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2021-02-08T07:06:44.703 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2021-02-08T07:06:44.917 INFO:teuthology.orchestra.run.smithi017.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-08T07:06:44.917 INFO:teuthology.orchestra.run.smithi017.stdout:0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded. 2021-02-08T07:06:44.919 DEBUG:teuthology.orchestra.run.smithi017:> dpkg -s linux-image-generic 2021-02-08T07:06:44.938 INFO:teuthology.orchestra.run.smithi017.stdout:Package: linux-image-generic 2021-02-08T07:06:44.938 INFO:teuthology.orchestra.run.smithi017.stdout:Status: install ok installed 2021-02-08T07:06:44.938 INFO:teuthology.orchestra.run.smithi017.stdout:Priority: optional 2021-02-08T07:06:44.938 INFO:teuthology.orchestra.run.smithi017.stdout:Section: kernel 2021-02-08T07:06:44.938 INFO:teuthology.orchestra.run.smithi017.stdout:Installed-Size: 17 2021-02-08T07:06:44.938 INFO:teuthology.orchestra.run.smithi017.stdout:Maintainer: Ubuntu Kernel Team 2021-02-08T07:06:44.938 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: amd64 2021-02-08T07:06:44.938 INFO:teuthology.orchestra.run.smithi017.stdout:Source: linux-meta 2021-02-08T07:06:44.939 INFO:teuthology.orchestra.run.smithi017.stdout:Version: 4.15.0.135.122 2021-02-08T07:06:44.939 INFO:teuthology.orchestra.run.smithi017.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-08T07:06:44.939 INFO:teuthology.orchestra.run.smithi017.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-08T07:06:44.939 INFO:teuthology.orchestra.run.smithi017.stdout:Recommends: thermald 2021-02-08T07:06:44.939 INFO:teuthology.orchestra.run.smithi017.stdout:Description: Generic Linux kernel image 2021-02-08T07:06:44.939 INFO:teuthology.orchestra.run.smithi017.stdout: This package will always depend on the latest generic kernel image 2021-02-08T07:06:44.939 INFO:teuthology.orchestra.run.smithi017.stdout: available. 2021-02-08T07:06:44.940 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2021-02-08T07:06:44.987 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.QqYtzxG6J6 2021-02-08T07:06:44.988 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QqYtzxG6J6 2021-02-08T07:06:45.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.QqYtzxG6J6 2021-02-08T07:06:45.176 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.QqYtzxG6J6 is 8KB 2021-02-08T07:06:45.232 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.QqYtzxG6J6 2021-02-08T07:06:45.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-08T07:06:45.293 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:06:45.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-08T07:06:45.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-08T07:06:45.377 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-08T07:06:45.377 DEBUG:teuthology.orchestra.run.smithi017:> sudo update-grub 2021-02-08T07:06:46.644 INFO:teuthology.orchestra.run.smithi017.stderr:Sourcing file `/etc/default/grub' 2021-02-08T07:06:46.662 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2021-02-08T07:06:46.885 INFO:teuthology.orchestra.run.smithi017.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-08T07:06:46.897 INFO:teuthology.orchestra.run.smithi017.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-08T07:06:47.172 INFO:teuthology.orchestra.run.smithi017.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-08T07:06:47.184 INFO:teuthology.orchestra.run.smithi017.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-08T07:06:48.392 INFO:teuthology.orchestra.run.smithi017.stderr:done 2021-02-08T07:06:48.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2021-02-08T07:06:48.395 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2021-02-08T07:06:48.395 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2021-02-08T07:06:48.395 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2021-02-08T07:06:48.442 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2021-02-08T07:06:48.692 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-08T07:06:48.692 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-08T07:06:48.761 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-08T07:06:48.842 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-08T07:06:50.570 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2021-02-08T07:06:50.620 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-08T07:06:50.703 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2021-02-08T07:06:50.942 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2021-02-08T07:06:50.942 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2021-02-08T07:06:51.023 INFO:teuthology.orchestra.run.smithi110.stderr:E: Unable to locate package linux-image-current-generic 2021-02-08T07:06:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-08T07:06:51.024 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-08T07:06:51.098 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2021-02-08T07:06:51.328 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2021-02-08T07:06:51.328 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2021-02-08T07:06:51.525 INFO:teuthology.orchestra.run.smithi110.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-08T07:06:51.525 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded. 2021-02-08T07:06:51.527 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2021-02-08T07:06:51.588 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 17 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 4.15.0.135.122 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2021-02-08T07:06:51.589 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2021-02-08T07:06:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2021-02-08T07:06:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2021-02-08T07:06:51.591 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2021-02-08T07:06:51.636 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.c0jmi9YZxb 2021-02-08T07:06:51.637 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub/grub.cfg /tmp/tmp.c0jmi9YZxb 2021-02-08T07:06:51.691 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.c0jmi9YZxb 2021-02-08T07:06:51.807 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.c0jmi9YZxb is 8KB 2021-02-08T07:06:51.818 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.c0jmi9YZxb 2021-02-08T07:06:51.825 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-08T07:06:51.881 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:06:51.881 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-08T07:06:51.881 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-08T07:06:51.950 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-08T07:06:51.950 DEBUG:teuthology.orchestra.run.smithi110:> sudo update-grub 2021-02-08T07:06:53.364 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub' 2021-02-08T07:06:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2021-02-08T07:06:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-08T07:06:53.735 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-08T07:06:54.084 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-08T07:06:54.097 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-08T07:06:55.095 INFO:teuthology.orchestra.run.smithi110.stderr:done 2021-02-08T07:06:55.096 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2021-02-08T07:07:25.126 INFO:teuthology.misc:Re-opening connections... 2021-02-08T07:07:25.127 INFO:teuthology.misc:trying to connect to ubuntu@smithi017.front.sepia.ceph.com 2021-02-08T07:07:25.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-08T07:07:25.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-08T07:07:43.682 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2021-02-08T07:07:43.683 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2021-02-08T07:07:43.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-08T07:07:43.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-08T07:08:43.686 DEBUG:teuthology.orchestra.remote:timed out 2021-02-08T07:08:43.686 DEBUG:teuthology.misc:waited 78.55966758728027 2021-02-08T07:08:44.687 INFO:teuthology.misc:trying to connect to ubuntu@smithi017.front.sepia.ceph.com 2021-02-08T07:08:44.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-08T07:08:44.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-08T07:08:44.765 DEBUG:teuthology.orchestra.run.smithi017:> true 2021-02-08T07:08:45.188 DEBUG:teuthology.misc:waited 80.06157970428467 2021-02-08T07:08:46.189 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2021-02-08T07:08:46.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-08T07:08:46.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-08T07:08:46.266 DEBUG:teuthology.orchestra.run.smithi110:> true 2021-02-08T07:08:46.718 DEBUG:teuthology.misc:waited 81.59157061576843 2021-02-08T07:08:47.719 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2021-02-08T07:08:47.719 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-135-generic"... 2021-02-08T07:08:47.719 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2021-02-08T07:08:47.727 INFO:teuthology.orchestra.run.smithi017.stdout:4.15.0-135-generic 2021-02-08T07:08:47.727 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-08T07:08:47.727 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-08T07:08:47.727 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-08T07:08:47.727 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2021-02-08T07:08:47.727 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-135-generic"... 2021-02-08T07:08:47.728 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2021-02-08T07:08:47.734 INFO:teuthology.orchestra.run.smithi110.stdout:4.15.0-135-generic 2021-02-08T07:08:47.734 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-08T07:08:47.734 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-08T07:08:47.734 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-08T07:08:48.735 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-02-08T07:08:48.736 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-08T07:08:48.781 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2021-02-08T07:08:48.804 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-02-08T07:08:48.804 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-08T07:08:48.834 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2021-02-08T07:08:48.856 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-08T07:08:48.866 INFO:teuthology.task.internal:Creating test directory... 2021-02-08T07:08:48.866 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-08T07:08:48.868 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-08T07:08:48.904 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-08T07:08:48.914 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-08T07:08:48.927 INFO:teuthology.task.internal:Creating archive directory... 2021-02-08T07:08:48.928 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-08T07:08:48.930 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-08T07:08:48.966 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-08T07:08:48.976 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-08T07:08:48.976 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-08T07:08:48.980 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-08T07:08:49.002 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-08T07:08:49.011 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-08T07:08:49.020 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-08T07:08:49.032 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-08T07:08:49.033 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-08T07:08:49.043 INFO:teuthology.task.internal:Configuring sudo... 2021-02-08T07:08:49.043 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-08T07:08:49.056 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-08T07:08:49.089 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-08T07:08:49.100 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-08T07:08:49.101 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-08T07:08:49.111 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-08T07:08:49.136 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-08T07:08:49.164 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-08T07:08:49.212 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:08:49.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-08T07:08:49.276 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-08T07:08:49.283 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-08T07:08:49.333 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:08:49.333 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-08T07:08:49.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2021-02-08T07:08:49.403 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2021-02-08T07:08:49.486 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-08T07:08:49.501 INFO:teuthology.task.internal:Starting timer... 2021-02-08T07:08:49.501 INFO:teuthology.run_tasks:Running task pcp... 2021-02-08T07:08:49.515 INFO:teuthology.run_tasks:Running task selinux... 2021-02-08T07:08:49.525 DEBUG:teuthology.task.selinux:Excluding smithi017: OS 'ubuntu' does not support SELinux 2021-02-08T07:08:49.525 DEBUG:teuthology.task.selinux:Excluding smithi110: OS 'ubuntu' does not support SELinux 2021-02-08T07:08:49.526 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-08T07:08:49.526 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-02-08T07:08:49.526 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-08T07:08:49.526 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-08T07:08:49.538 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-02-08T07:08:49.539 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-08T07:08:49.559 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-08T07:08:49.562 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-08T07:08:49.907 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-08T07:08:50.801 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-08T07:08:57.628 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* Monday 08 February 2021 07:08:53 +0000 (0:00:00.072) 0:00:00.072 ******* Monday 08 February 2021 07:08:53 +0000 (0:00:00.014) 0:00:00.087 ******* TASK [Gathering Facts] ********************************************************* ok: [smithi110.front.sepia.ceph.com] Monday 08 February 2021 07:08:55 +0000 (0:00:01.718) 0:00:01.805 ******* ok: [smithi017.front.sepia.ceph.com] Monday 08 February 2021 07:08:55 +0000 (0:00:00.087) 0:00:01.892 ******* TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} Monday 08 February 2021 07:08:55 +0000 (0:00:00.209) 0:00:02.101 ******* Monday 08 February 2021 07:08:55 +0000 (0:00:00.033) 0:00:02.134 ******* TASK [ansible-managed : Create the ansible user.] ****************************** changed: [smithi110.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Monday 08 February 2021 07:08:55 +0000 (0:00:00.429) 0:00:02.564 ******* changed: [smithi017.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Monday 08 February 2021 07:08:55 +0000 (0:00:00.044) 0:00:02.608 ******* TASK [ansible-managed : Delete the ansible users password.] ******************** changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.06 2021-02-08T07:08:57.630 INFO:teuthology.task.ansible.out:0077", "end": "2021-02-08 07:08:56.555197", "rc": 0, "start": "2021-02-08 07:08:56.495120", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.069060", "end": "2021-02-08 07:08:56.165186", "rc": 0, "start": "2021-02-08 07:08:56.096126", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} Monday 08 February 2021 07:08:56 +0000 (0:00:00.304) 0:00:02.912 ******* Monday 08 February 2021 07:08:56 +0000 (0:00:00.032) 0:00:02.945 ******* TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Monday 08 February 2021 07:08:56 +0000 (0:00:00.251) 0:00:03.196 ******* Monday 08 February 2021 07:08:56 +0000 (0:00:00.033) 0:00:03.230 ******* TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612768136.5512505-241911941718919/source", "state": "file", "uid": 0} changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp- 2021-02-08T07:08:57.631 INFO:teuthology.task.ansible.out:1612768136.5884671-195791134057651/source", "state": "file", "uid": 0} Monday 08 February 2021 07:08:57 +0000 (0:00:00.648) 0:00:03.878 ******* Monday 08 February 2021 07:08:57 +0000 (0:00:00.041) 0:00:03.919 ******* TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR 2021-02-08T07:08:57.632 INFO:teuthology.task.ansible.out:9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j 2021-02-08T07:08:57.634 INFO:teuthology.task.ansible.out:0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8 2021-02-08T07:08:57.635 INFO:teuthology.task.ansible.out:Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-08T07:08:57.698 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-02-08T07:08:57.705 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-02-08T07:08:57.856 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:57 +0000 (0:00:00.595) 0:00:04.514 ******* 2021-02-08T07:08:57.892 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:57 +0000 (0:00:00.035) 0:00:04.550 ******* 2021-02-08T07:08:58.582 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi110.front.sepia.ceph.com] 2021-02-08T07:08:58.630 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] 2021-02-08T07:08:58.699 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-02-08T07:08:58.700 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:58 +0000 (0:00:00.806) 0:00:05.357 ******* 2021-02-08T07:08:58.747 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:58 +0000 (0:00:00.048) 0:00:05.405 ******* 2021-02-08T07:08:59.054 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-02-08T07:08:59.056 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.157451", "end": "2021-02-08 07:08:59.028629", "rc": 0, "start": "2021-02-08 07:08:58.871178", "stderr": "", "stderr_lines": [], "stdout": "2470ca37c45480b983e40a390839e1c5ce065bc7", "stdout_lines": ["2470ca37c45480b983e40a390839e1c5ce065bc7"]} 2021-02-08T07:08:59.092 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.345) 0:00:05.750 ******* 2021-02-08T07:08:59.099 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.148984", "end": "2021-02-08 07:08:59.067593", "rc": 0, "start": "2021-02-08 07:08:58.918609", "stderr": "", "stderr_lines": [], "stdout": "2470ca37c45480b983e40a390839e1c5ce065bc7", "stdout_lines": ["2470ca37c45480b983e40a390839e1c5ce065bc7"]} 2021-02-08T07:08:59.138 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.045) 0:00:05.796 ******* 2021-02-08T07:08:59.228 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-08T07:08:59.230 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.004451", "end": "2021-02-08 07:08:59.559676", "failed_when_result": false, "rc": 0, "start": "2021-02-08 07:08:59.555225", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} 2021-02-08T07:08:59.266 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.127) 0:00:05.924 ******* 2021-02-08T07:08:59.283 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011541", "end": "2021-02-08 07:08:59.223804", "failed_when_result": false, "rc": 0, "start": "2021-02-08 07:08:59.212263", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} 2021-02-08T07:08:59.319 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.052) 0:00:05.976 ******* 2021-02-08T07:08:59.325 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-02-08T07:08:59.326 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.361 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.042) 0:00:06.019 ******* 2021-02-08T07:08:59.368 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.405 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.044) 0:00:06.063 ******* 2021-02-08T07:08:59.411 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2021-02-08T07:08:59.413 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.450 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.045) 0:00:06.108 ******* 2021-02-08T07:08:59.456 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.494 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.043) 0:00:06.152 ******* 2021-02-08T07:08:59.502 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-02-08T07:08:59.503 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.539 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.045) 0:00:06.197 ******* 2021-02-08T07:08:59.546 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.583 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.043) 0:00:06.241 ******* 2021-02-08T07:08:59.590 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2021-02-08T07:08:59.591 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.628 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.044) 0:00:06.286 ******* 2021-02-08T07:08:59.635 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.672 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.043) 0:00:06.330 ******* 2021-02-08T07:08:59.681 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2021-02-08T07:08:59.682 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.721 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.049) 0:00:06.379 ******* 2021-02-08T07:08:59.728 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:08:59.769 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.047) 0:00:06.427 ******* 2021-02-08T07:08:59.870 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-02-08T07:08:59.871 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-08T07:08:59.910 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.140) 0:00:06.568 ******* 2021-02-08T07:08:59.916 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-08T07:08:59.956 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:08:59 +0000 (0:00:00.046) 0:00:06.614 ******* 2021-02-08T07:09:00.087 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-08T07:09:00.089 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-08T07:09:00.131 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-08T07:09:00.226 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-08T07:09:00.269 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-08T07:09:00.375 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-08T07:09:00.397 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-08T07:09:00.399 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:00.515 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-08T07:09:00.527 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-08T07:09:00.656 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-08T07:09:00.668 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-08T07:09:00.784 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-08T07:09:00.806 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-08T07:09:00.911 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-08T07:09:00.954 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-08T07:09:01.039 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-08T07:09:01.092 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-08T07:09:01.170 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-08T07:09:01.234 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-08T07:09:01.298 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-08T07:09:01.361 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-08T07:09:01.436 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-08T07:09:01.499 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-08T07:09:01.563 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-08T07:09:01.564 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:01.637 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-08T07:09:01.691 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-08T07:09:01.765 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-08T07:09:01.829 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-08T07:09:01.892 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-08T07:09:01.956 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-08T07:09:02.030 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-08T07:09:02.084 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-08T07:09:02.086 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:02.168 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-08T07:09:02.211 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-08T07:09:02.315 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-08T07:09:02.317 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:02.348 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-08T07:09:02.452 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-08T07:09:02.484 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-08T07:09:02.600 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-08T07:09:02.622 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-08T07:09:02.728 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-08T07:09:02.751 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-08T07:09:02.857 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-08T07:09:02.879 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-08T07:09:02.997 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-08T07:09:03.010 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-08T07:09:03.128 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-08T07:09:03.140 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-08T07:09:03.271 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-08T07:09:03.304 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-08T07:09:03.390 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-08T07:09:03.454 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-08T07:09:03.528 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-08T07:09:03.602 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-08T07:09:03.645 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-08T07:09:03.771 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-08T07:09:03.772 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-08T07:09:03.908 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-08T07:09:03.909 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-08T07:09:04.035 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-08T07:09:04.046 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-08T07:09:04.164 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-08T07:09:04.199 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-08T07:09:04.294 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-08T07:09:04.336 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-08T07:09:04.420 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-08T07:09:04.484 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-08T07:09:04.486 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:04.540 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-08T07:09:04.635 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-08T07:09:04.668 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-08T07:09:04.784 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-08T07:09:04.806 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-08T07:09:04.932 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-08T07:09:04.933 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-08T07:09:05.059 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-08T07:09:05.060 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-08T07:09:05.196 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-08T07:09:05.197 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-08T07:09:05.314 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-08T07:09:05.335 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-08T07:09:05.451 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-08T07:09:05.452 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:05.484 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-08T07:09:05.569 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-08T07:09:05.635 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-08T07:09:05.699 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-08T07:09:05.773 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-08T07:09:05.837 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-08T07:09:05.911 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-08T07:09:05.965 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-08T07:09:06.039 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-08T07:09:06.093 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-08T07:09:06.177 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-08T07:09:06.220 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-08T07:09:06.315 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-08T07:09:06.360 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-08T07:09:06.465 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-08T07:09:06.488 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-08T07:09:06.593 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-08T07:09:06.615 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-08T07:09:06.731 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-08T07:09:06.743 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-08T07:09:06.847 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-08T07:09:06.880 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-08T07:09:06.998 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-08T07:09:07.025 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-08T07:09:07.157 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-08T07:09:07.159 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:07.209 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-08T07:09:07.304 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-08T07:09:07.380 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-08T07:09:07.453 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-08T07:09:07.527 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-08T07:09:07.580 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-08T07:09:07.676 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-08T07:09:07.708 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-08T07:09:07.817 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-08T07:09:07.840 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-08T07:09:07.967 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-08T07:09:07.979 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-08T07:09:08.107 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-08T07:09:08.109 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:08.120 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-08T07:09:08.235 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-08T07:09:08.237 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:08.259 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-08T07:09:08.367 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-08T07:09:08.412 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-08T07:09:08.498 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-08T07:09:08.552 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-08T07:09:08.627 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-08T07:09:08.691 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-08T07:09:08.756 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-08T07:09:08.843 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-08T07:09:08.888 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-08T07:09:08.994 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-08T07:09:09.016 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-08T07:09:09.142 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-08T07:09:09.144 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-08T07:09:09.270 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-08T07:09:09.292 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-08T07:09:09.398 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-08T07:09:09.430 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-08T07:09:09.526 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-08T07:09:09.580 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-08T07:09:09.653 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-08T07:09:09.728 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-08T07:09:09.792 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-08T07:09:09.846 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-08T07:09:09.920 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-08T07:09:09.994 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-08T07:09:10.047 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-08T07:09:10.131 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-08T07:09:10.184 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-08T07:09:10.271 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-08T07:09:10.303 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-08T07:09:10.409 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-08T07:09:10.436 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-08T07:09:10.550 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-08T07:09:10.593 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-08T07:09:10.678 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-08T07:09:10.721 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-08T07:09:10.816 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-08T07:09:10.848 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-08T07:09:10.954 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-08T07:09:10.988 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-08T07:09:11.095 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-08T07:09:11.117 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-08T07:09:11.243 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-08T07:09:11.256 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-08T07:09:11.373 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-08T07:09:11.385 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-08T07:09:11.511 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-08T07:09:11.512 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-08T07:09:11.637 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-08T07:09:11.659 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-08T07:09:11.777 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-08T07:09:11.809 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-08T07:09:11.904 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-08T07:09:11.926 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-08T07:09:12.023 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-08T07:09:12.055 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-08T07:09:12.161 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-08T07:09:12.205 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-08T07:09:12.313 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-08T07:09:12.388 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-08T07:09:12.465 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-08T07:09:12.574 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-08T07:09:12.618 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-08T07:09:12.758 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-08T07:09:12.769 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-08T07:09:12.912 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-08T07:09:12.914 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-08T07:09:13.052 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-08T07:09:13.066 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-08T07:09:13.201 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-08T07:09:13.216 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-08T07:09:13.332 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-08T07:09:13.334 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-08T07:09:13.338 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:13.468 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-08T07:09:13.490 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-08T07:09:13.606 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-08T07:09:13.629 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-08T07:09:13.735 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-08T07:09:13.778 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-08T07:09:13.890 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-08T07:09:13.915 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-08T07:09:14.021 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-08T07:09:14.043 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-08T07:09:14.165 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-08T07:09:14.198 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-08T07:09:14.303 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-08T07:09:14.337 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-08T07:09:14.432 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-08T07:09:14.485 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-08T07:09:14.572 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-08T07:09:14.615 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-08T07:09:14.690 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-08T07:09:14.765 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-08T07:09:14.819 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-08T07:09:14.895 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-08T07:09:14.949 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-08T07:09:15.044 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-08T07:09:15.087 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-08T07:09:15.172 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-08T07:09:15.215 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-08T07:09:15.314 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-08T07:09:15.347 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-08T07:09:15.464 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-08T07:09:15.475 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-08T07:09:15.592 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-08T07:09:15.603 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-08T07:09:15.719 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-08T07:09:15.732 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-08T07:09:15.848 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-08T07:09:15.870 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-08T07:09:15.986 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-08T07:09:15.997 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-08T07:09:16.113 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-08T07:09:16.124 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-08T07:09:16.240 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-08T07:09:16.251 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-08T07:09:16.366 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-08T07:09:16.399 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-08T07:09:16.506 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-08T07:09:16.539 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-08T07:09:16.634 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-08T07:09:16.667 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-08T07:09:16.761 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-08T07:09:16.814 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-08T07:09:16.889 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-08T07:09:16.963 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-08T07:09:17.027 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-08T07:09:17.090 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-08T07:09:17.155 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-08T07:09:17.243 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-08T07:09:17.289 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-08T07:09:17.374 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-08T07:09:17.406 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-08T07:09:17.522 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-08T07:09:17.544 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-08T07:09:17.661 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-08T07:09:17.672 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-08T07:09:17.778 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-08T07:09:17.790 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-08T07:09:17.917 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-08T07:09:17.928 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-08T07:09:18.059 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-08T07:09:18.066 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-08T07:09:18.182 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-02-08T07:09:18.193 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-08T07:09:18.307 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:18 +0000 (0:00:18.350) 0:00:24.965 ******* 2021-02-08T07:09:18.329 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-08T07:09:18.340 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-08T07:09:18.378 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:18 +0000 (0:00:00.070) 0:00:25.036 ******* 2021-02-08T07:09:18.427 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-08T07:09:18.429 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-08T07:09:18.430 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-08T07:09:18.432 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-08T07:09:18.433 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-08T07:09:18.434 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-08T07:09:18.436 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-08T07:09:18.437 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 2021-02-08T07:09:18.437 INFO:teuthology.task.ansible.out:5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-08T07:09:18.438 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-08T07:09:18.440 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-08T07:09:18.441 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9 2021-02-08T07:09:18.442 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-08T07:09:18.444 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-08T07:09:18.445 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395 2021-02-08T07:09:18.445 INFO:teuthology.task.ansible.out:862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}]}, "changed": false} 2021-02-08T07:09:18.490 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:18 +0000 (0:00:00.111) 0:00:25.147 ******* 2021-02-08T07:09:18.498 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-08T07:09:18.499 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-08T07:09:18.585 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-08T07:09:18.734 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-08T07:09:18.861 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-08T07:09:18.988 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-08T07:09:19.138 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-08T07:09:19.292 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-08T07:09:19.345 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-08T07:09:19.347 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com -> localhost] => {"after": "2470ca37c45480b983e40a390839e1c5ce065bc7", "attempts": 1, "before": "2470ca37c45480b983e40a390839e1c5ce065bc7", "changed": false, "remote_url_changed": false} 2021-02-08T07:09:19.382 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:19 +0000 (0:00:00.892) 0:00:26.040 ******* 2021-02-08T07:09:19.431 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-08T07:09:19.433 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-08T07:09:19.568 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:19.570 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a 2021-02-08T07:09:19.571 INFO:teuthology.task.ansible.out:09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-02-08T07:09:19.707 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:19 +0000 (0:00:00.324) 0:00:26.364 ******* 2021-02-08T07:09:19.715 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:19.716 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-08T07:09:19.730 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-08T07:09:19.766 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:19 +0000 (0:00:00.059) 0:00:26.423 ******* 2021-02-08T07:09:19.816 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-08T07:09:19.818 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-08T07:09:19.819 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-08T07:09:19.821 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-08T07:09:19.822 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-08T07:09:19.823 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-08T07:09:19.825 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-08T07:09:19.826 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-08T07:09:19.827 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-08T07:09:19.829 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-08T07:09:19.830 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9 2021-02-08T07:09:19.831 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-08T07:09:19.833 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "ac 2021-02-08T07:09:19.833 INFO:teuthology.task.ansible.out:lewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-08T07:09:19.834 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}]}, "changed": false} 2021-02-08T07:09:19.836 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:19.837 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-08T07:09:19.879 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:19 +0000 (0:00:00.113) 0:00:26.537 ******* 2021-02-08T07:09:19.971 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-08T07:09:19.973 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-08T07:09:20.099 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-08T07:09:20.226 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-08T07:09:20.228 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-08T07:09:20.355 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-08T07:09:20.439 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-08T07:09:20.441 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com -> localhost] => {"after": "2470ca37c45480b983e40a390839e1c5ce065bc7", "attempts": 1, "before": "2470ca37c45480b983e40a390839e1c5ce065bc7", "changed": false, "remote_url_changed": false} 2021-02-08T07:09:20.476 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:20 +0000 (0:00:00.596) 0:00:27.134 ******* 2021-02-08T07:09:20.493 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:20.494 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-08T07:09:20.620 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-08T07:09:20.620 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:20.758 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-08T07:09:20.759 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-08T07:09:20.779 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-08T07:09:20.886 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-08T07:09:20.888 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-08T07:09:20.897 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:20.898 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:20.951 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-08T07:09:21.025 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-08T07:09:21.098 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-08T07:09:21.162 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-08T07:09:21.164 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-08T07:09:21.257 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-08T07:09:21.259 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-08T07:09:21.299 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-08T07:09:21.384 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-08T07:09:21.429 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-08T07:09:21.550 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-08T07:09:21.583 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-08T07:09:21.585 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-08T07:09:21.680 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-08T07:09:21.723 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-08T07:09:21.807 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-08T07:09:21.861 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-08T07:09:21.935 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-08T07:09:21.937 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-08T07:09:21.999 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-08T07:09:22.021 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:22.063 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-08T07:09:22.065 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-08T07:09:22.162 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-08T07:09:22.163 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-08T07:09:22.195 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-08T07:09:22.299 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-08T07:09:22.301 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-08T07:09:22.311 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIH 2021-02-08T07:09:22.313 INFO:teuthology.task.ansible.out:Dvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-08T07:09:22.322 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:22.448 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-08T07:09:22.450 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-08T07:09:22.460 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-08T07:09:22.567 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-08T07:09:22.599 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-08T07:09:22.600 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-08T07:09:22.694 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-08T07:09:22.736 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-08T07:09:22.821 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-08T07:09:22.864 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-08T07:09:22.949 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-08T07:09:23.002 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-08T07:09:23.118 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-08T07:09:23.140 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-08T07:09:23.246 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-08T07:09:23.248 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-08T07:09:23.289 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-08T07:09:23.380 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-08T07:09:23.444 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-08T07:09:23.446 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-08T07:09:23.518 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-08T07:09:23.582 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-08T07:09:23.584 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-08T07:09:23.647 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-08T07:09:23.712 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-08T07:09:23.766 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-08T07:09:23.841 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-08T07:09:23.894 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-08T07:09:23.981 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-08T07:09:23.982 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-08T07:09:24.024 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-08T07:09:24.118 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-08T07:09:24.148 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-08T07:09:24.245 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-08T07:09:24.288 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-08T07:09:24.383 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-08T07:09:24.425 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-08T07:09:24.529 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-08T07:09:24.561 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-08T07:09:24.562 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-08T07:09:24.686 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-08T07:09:24.708 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-08T07:09:24.709 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-08T07:09:24.813 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-08T07:09:24.815 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-08T07:09:24.824 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-08T07:09:24.930 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-08T07:09:24.952 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-08T07:09:25.078 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-08T07:09:25.079 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-08T07:09:25.206 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-08T07:09:25.207 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-08T07:09:25.208 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-08T07:09:25.325 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-08T07:09:25.348 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-08T07:09:25.349 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-08T07:09:25.350 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-08T07:09:25.453 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-08T07:09:25.475 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-08T07:09:25.591 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-08T07:09:25.602 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-08T07:09:25.718 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-08T07:09:25.740 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-08T07:09:25.741 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:25.857 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-08T07:09:25.868 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-08T07:09:25.965 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-08T07:09:25.999 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-08T07:09:26.105 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-08T07:09:26.106 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-08T07:09:26.126 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-08T07:09:26.127 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-08T07:09:26.241 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-08T07:09:26.243 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-08T07:09:26.253 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-08T07:09:26.368 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-08T07:09:26.390 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-08T07:09:26.505 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-08T07:09:26.516 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-08T07:09:26.518 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-08T07:09:26.632 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-08T07:09:26.643 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-08T07:09:26.769 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-08T07:09:26.770 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-08T07:09:26.771 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-08T07:09:26.895 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-08T07:09:26.896 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-08T07:09:26.917 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-08T07:09:26.919 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-08T07:09:26.920 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-08T07:09:27.024 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-08T07:09:27.056 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-08T07:09:27.151 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-08T07:09:27.162 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-08T07:09:27.267 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-08T07:09:27.309 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-08T07:09:27.394 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-08T07:09:27.436 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-08T07:09:27.530 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-08T07:09:27.573 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-08T07:09:27.658 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-08T07:09:27.723 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-08T07:09:27.724 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-08T07:09:27.788 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-08T07:09:27.790 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:27.864 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-08T07:09:27.920 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-08T07:09:28.005 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-08T07:09:28.038 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-08T07:09:28.154 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-08T07:09:28.155 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-08T07:09:28.165 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-08T07:09:28.292 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-08T07:09:28.293 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-08T07:09:28.430 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-08T07:09:28.431 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-08T07:09:28.556 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-08T07:09:28.578 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-08T07:09:28.579 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-08T07:09:28.682 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-08T07:09:28.715 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-08T07:09:28.810 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-08T07:09:28.862 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-08T07:09:28.926 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-08T07:09:29.012 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-08T07:09:30.029 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4 2021-02-08T07:09:30.031 INFO:teuthology.task.ansible.out:FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authori 2021-02-08T07:09:30.032 INFO:teuthology.task.ansible.out:zed_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI3 2021-02-08T07:09:30.033 INFO:teuthology.task.ansible.out:9OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYi 2021-02-08T07:09:30.035 INFO:teuthology.task.ansible.out:SQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFm 2021-02-08T07:09:30.036 INFO:teuthology.task.ansible.out:FUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": 2021-02-08T07:09:30.037 INFO:teuthology.task.ansible.out: "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-08T07:09:30.038 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:30.039 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-08T07:09:30.040 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-08T07:09:30.041 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-08T07:09:30.062 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-08T07:09:30.095 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-08T07:09:30.209 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-08T07:09:30.221 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-02-08T07:09:30.222 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-08T07:09:30.346 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-08T07:09:30.347 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-02-08T07:09:30.348 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-08T07:09:30.472 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-08T07:09:30.472 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-08T07:09:30.599 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-08T07:09:30.610 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-08T07:09:30.727 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-08T07:09:30.738 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-08T07:09:30.843 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-08T07:09:30.875 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-08T07:09:30.970 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-08T07:09:30.992 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-08T07:09:31.098 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-08T07:09:31.130 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-08T07:09:31.225 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-08T07:09:31.246 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-08T07:09:31.341 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-08T07:09:31.342 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-08T07:09:31.394 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-08T07:09:31.468 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-08T07:09:31.521 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-02-08T07:09:31.522 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-08T07:09:31.595 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-08T07:09:31.658 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-08T07:09:31.725 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-08T07:09:31.800 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-08T07:09:31.853 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-08T07:09:31.927 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-08T07:09:31.980 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-08T07:09:32.075 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-08T07:09:32.097 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-08T07:09:32.214 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-02-08T07:09:32.215 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-08T07:09:32.246 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-02-08T07:09:32.248 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-08T07:09:32.352 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-02-08T07:09:32.353 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-08T07:09:32.374 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-08T07:09:32.501 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-08T07:09:32.502 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-08T07:09:32.644 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-08T07:09:32.666 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-08T07:09:32.783 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-08T07:09:32.826 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-08T07:09:32.921 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-08T07:09:32.953 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-08T07:09:33.039 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-08T07:09:33.113 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-08T07:09:33.167 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-08T07:09:33.230 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-08T07:09:33.294 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-08T07:09:33.368 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-08T07:09:33.421 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-08T07:09:33.495 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-08T07:09:33.497 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-08T07:09:33.549 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-08T07:09:33.623 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-08T07:09:33.676 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-08T07:09:33.740 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-08T07:09:33.804 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-08T07:09:33.877 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-08T07:09:33.931 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-08T07:09:34.015 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-08T07:09:34.058 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-08T07:09:34.150 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-08T07:09:34.185 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-08T07:09:34.301 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-08T07:09:34.312 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-08T07:09:34.451 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-08T07:09:34.452 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-02-08T07:09:34.453 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-08T07:09:34.579 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-08T07:09:34.601 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-08T07:09:34.706 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-02-08T07:09:34.707 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-08T07:09:34.718 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-08T07:09:34.845 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-08T07:09:34.866 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-08T07:09:34.972 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-08T07:09:34.994 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-08T07:09:35.090 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-08T07:09:35.134 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-08T07:09:35.218 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-08T07:09:35.250 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-08T07:09:35.356 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-08T07:09:35.377 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-08T07:09:35.489 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-08T07:09:35.543 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-08T07:09:35.630 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-08T07:09:35.707 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-08T07:09:35.774 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-08T07:09:35.860 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-08T07:09:35.914 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-08T07:09:36.010 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-08T07:09:36.063 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-08T07:09:36.169 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-08T07:09:36.201 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-08T07:09:36.338 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-08T07:09:36.360 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-08T07:09:36.478 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-08T07:09:36.532 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-08T07:09:36.651 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-08T07:09:36.682 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-08T07:09:36.808 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-08T07:09:36.821 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-08T07:09:36.962 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-08T07:09:36.975 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-08T07:09:37.118 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-08T07:09:37.143 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-08T07:09:37.238 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:37 +0000 (0:00:16.762) 0:00:43.896 ******* 2021-02-08T07:09:37.266 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-02-08T07:09:37.268 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-08T07:09:37.298 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:37.300 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.309 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.320 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.331 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.333 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.343 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.354 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.354 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.366 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.399 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:37.402 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-08T07:09:37.506 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:37.507 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2021-02-08T07:09:37.509 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.518 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.551 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:37.551 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-08T07:09:37.647 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:37.649 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-08T07:09:37.711 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:37.713 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-08T07:09:37.787 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:37.788 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfad 2021-02-08T07:09:37.790 INFO:teuthology.task.ansible.out:a28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.799 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.810 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.820 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.821 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.832 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.844 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys 2021-02-08T07:09:37.845 INFO:teuthology.task.ansible.out:", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-08T07:09:37.854 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.855 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.866 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.877 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.889 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.890 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.901 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.912 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.924 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.936 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.947 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.959 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.959 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.971 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:37.972 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.983 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:37.995 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.008 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.008 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.020 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.031 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.032 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.043 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.054 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.066 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.067 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.077 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.089 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.100 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.101 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.113 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.124 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.125 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.135 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.146 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.157 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.159 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.168 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.179 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.190 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.191 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.202 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.214 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.225 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.226 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:38.230 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.238 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.239 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:38.250 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.261 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.272 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.273 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:38.283 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.285 INFO:teuthology.task.ansible.out: skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.295 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.306 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.307 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.318 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.330 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.330 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.342 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.343 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:38.353 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.355 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-08T07:09:38.355 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:38.355 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.366 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.378 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.380 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.400 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.401 INFO:teuthology.task.ansible.out: skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.412 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.424 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.425 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.435 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.446 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.448 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.458 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.469 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.470 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.481 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.493 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:38.494 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.505 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.515 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.516 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.527 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.538 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.539 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.550 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.561 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.562 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.572 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.583 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.584 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.595 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.606 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.607 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.617 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.628 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.630 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-08T07:09:38.631 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.642 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.654 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.654 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.666 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.677 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.688 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.699 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.700 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.711 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.721 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.722 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.735 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.737 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.747 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.759 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:38.760 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.771 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.782 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.783 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.837 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:38 +0000 (0:00:01.599) 0:00:45.495 ******* 2021-02-08T07:09:38.865 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-08T07:09:38.867 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:38.903 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:38 +0000 (0:00:00.065) 0:00:45.561 ******* 2021-02-08T07:09:38.910 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:38.911 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-08T07:09:39.027 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-08T07:09:39.059 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:39.060 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-08T07:09:39.157 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:39.158 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-08T07:09:39.190 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:39.191 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-08T07:09:39.286 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:39.287 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-08T07:09:39.318 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:39.319 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-08T07:09:39.413 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:39.414 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-08T07:09:39.435 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:39.436 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-08T07:09:39.530 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-08T07:09:39.532 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-08T07:09:39.607 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:39.608 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-08T07:09:39.649 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:39 +0000 (0:00:00.745) 0:00:46.307 ******* 2021-02-08T07:09:39.694 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-08T07:09:39.696 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:39.707 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:39.708 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.718 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.730 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.741 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.753 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.765 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.777 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.788 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.790 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:39.801 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.803 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:39.823 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:39.825 INFO:teuthology.task.ansible.out:TASK [users : Remove revoked users] ******************************************** ok: [smithi017.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-08T07:09:39.930 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:39.932 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-08T07:09:39.942 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:39.954 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:39.955 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-08T07:09:39.957 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.073 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:40.075 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": nul 2021-02-08T07:09:40.076 INFO:teuthology.task.ansible.out:l, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-08T07:09:40.190 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-08T07:09:40.201 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:40.203 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.213 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.225 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.236 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.246 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.247 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.258 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.269 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.280 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.290 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.301 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.302 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.313 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.314 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:40.314 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-08T07:09:40.325 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:40.326 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.337 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.348 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.358 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.369 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.380 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.381 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:40.382 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.393 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.404 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.415 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.416 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.437 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.439 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi017.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-08T07:09:40.450 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:40.451 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.461 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.463 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:40.483 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.485 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.495 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.506 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.507 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.518 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.540 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.541 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.542 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.553 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.565 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.565 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:40.565 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-08T07:09:40.576 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:40.578 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.589 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.599 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.610 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.621 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.632 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.633 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.633 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:40.654 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.656 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.666 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.678 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.688 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:40.690 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.700 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.712 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.723 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.734 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.745 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.756 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.757 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.768 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.779 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.779 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.790 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.801 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.812 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:40.814 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-08T07:09:40.824 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.824 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.825 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.836 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.836 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.847 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.859 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.859 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.871 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.882 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.884 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.894 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.905 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.905 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.916 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.928 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.939 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.940 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:40.942 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-08T07:09:40.942 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:40.943 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.954 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.965 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.966 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.976 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.978 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:40.988 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.989 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:40.999 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.011 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.013 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.022 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.034 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.035 INFO:teuthology.task.ansible.out: skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.046 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.057 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.068 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.070 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.080 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:41.081 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.096 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.098 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:41.110 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.113 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.123 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.124 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.136 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.148 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.149 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.160 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.172 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.173 INFO:teuthology.task.ansible.out: skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.185 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.197 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.197 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:41.198 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi017.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-08T07:09:41.200 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:41.200 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:41.200 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-08T07:09:41.201 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.212 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.213 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.224 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.236 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.240 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.247 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.299 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:41 +0000 (0:00:01.649) 0:00:47.957 ******* 2021-02-08T07:09:41.325 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:41.327 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-08T07:09:41.337 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-08T07:09:41.338 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:41.372 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:41 +0000 (0:00:00.072) 0:00:48.030 ******* 2021-02-08T07:09:41.451 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:41.453 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-08T07:09:41.526 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-08T07:09:41.580 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-08T07:09:41.643 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-08T07:09:41.728 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-08T07:09:41.781 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-08T07:09:41.845 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-08T07:09:41.919 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-08T07:09:41.920 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:41.962 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-08T07:09:41.963 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:42.048 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-08T07:09:42.091 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-08T07:09:42.165 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-08T07:09:42.166 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:42.218 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-08T07:09:42.220 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:42.293 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-08T07:09:42.335 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-08T07:09:42.409 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-08T07:09:42.411 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:42.463 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-08T07:09:42.536 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-08T07:09:42.578 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-08T07:09:42.652 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-08T07:09:42.716 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-08T07:09:42.782 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-08T07:09:42.835 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-08T07:09:42.899 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-08T07:09:42.963 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-08T07:09:43.026 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-08T07:09:43.090 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-08T07:09:43.143 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-08T07:09:43.206 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-08T07:09:43.260 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-08T07:09:43.334 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-08T07:09:43.387 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-08T07:09:43.451 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-08T07:09:43.504 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-08T07:09:43.588 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-08T07:09:43.631 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-08T07:09:43.706 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-08T07:09:43.769 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-08T07:09:43.897 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-08T07:09:44.024 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-08T07:09:44.151 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-08T07:09:44.172 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-08T07:09:44.299 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-08T07:09:44.300 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-08T07:09:44.426 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-08T07:09:44.427 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-08T07:09:44.542 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-08T07:09:44.554 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-08T07:09:44.670 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-08T07:09:44.671 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-08T07:09:44.797 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-08T07:09:44.797 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-08T07:09:44.914 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-08T07:09:44.925 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-08T07:09:45.053 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-08T07:09:45.056 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-08T07:09:45.162 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-08T07:09:45.174 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-08T07:09:45.280 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-08T07:09:45.302 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-08T07:09:45.409 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-08T07:09:45.466 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:45 +0000 (0:00:04.094) 0:00:52.124 ******* 2021-02-08T07:09:45.473 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-08T07:09:45.571 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-08T07:09:45.690 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-08T07:09:45.817 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "checksum": "6bd1cd53e59aef6d9a4a202ec6756843cde9d9e4", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "b908bc6436240ef2dee4e6bf56b67554", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612768185.492549-119794146407207/source", "state": "file", "uid": 0} 2021-02-08T07:09:45.854 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:45 +0000 (0:00:00.388) 0:00:52.512 ******* 2021-02-08T07:09:45.862 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:45.863 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-08T07:09:45.885 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-08T07:09:45.886 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => { "msg": "Host smithi017.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-08T07:09:45.921 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:45 +0000 (0:00:00.066) 0:00:52.579 ******* 2021-02-08T07:09:45.938 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:45.939 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-08T07:09:45.950 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-08T07:09:45.951 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-08T07:09:45.985 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:45 +0000 (0:00:00.064) 0:00:52.643 ******* 2021-02-08T07:09:46.013 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-08T07:09:46.014 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.050 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.065) 0:00:52.708 ******* 2021-02-08T07:09:46.058 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:46.058 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-08T07:09:46.070 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-08T07:09:46.071 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.105 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.054) 0:00:52.763 ******* 2021-02-08T07:09:46.131 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-08T07:09:46.133 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.168 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.062) 0:00:52.826 ******* 2021-02-08T07:09:46.184 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:46.186 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-08T07:09:46.196 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-08T07:09:46.197 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.232 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.063) 0:00:52.889 ******* 2021-02-08T07:09:46.259 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-08T07:09:46.260 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.295 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.063) 0:00:52.953 ******* 2021-02-08T07:09:46.321 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-08T07:09:46.323 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.359 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.064) 0:00:53.017 ******* 2021-02-08T07:09:46.386 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-08T07:09:46.388 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.423 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.063) 0:00:53.081 ******* 2021-02-08T07:09:46.451 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-08T07:09:46.453 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.490 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.067) 0:00:53.148 ******* 2021-02-08T07:09:46.529 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-08T07:09:46.530 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:46.566 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.076) 0:00:53.224 ******* 2021-02-08T07:09:46.593 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-08T07:09:46.594 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.629 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.062) 0:00:53.286 ******* 2021-02-08T07:09:46.656 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-08T07:09:46.658 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.693 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.064) 0:00:53.351 ******* 2021-02-08T07:09:46.699 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:46.700 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-08T07:09:46.711 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-08T07:09:46.713 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:46.748 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.055) 0:00:53.406 ******* 2021-02-08T07:09:46.775 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-08T07:09:46.776 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.811 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.062) 0:00:53.468 ******* 2021-02-08T07:09:46.817 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:46.818 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-08T07:09:46.840 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-08T07:09:46.841 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:46.879 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.068) 0:00:53.537 ******* 2021-02-08T07:09:46.917 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-08T07:09:46.918 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:46.953 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:46 +0000 (0:00:00.073) 0:00:53.611 ******* 2021-02-08T07:09:46.959 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:46.959 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-08T07:09:46.982 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:46.983 INFO:teuthology.task.ansible.out:TASK [common : List CDN releases available to system] ************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.017 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.064) 0:00:53.675 ******* 2021-02-08T07:09:47.044 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-08T07:09:47.045 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.087 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.069) 0:00:53.745 ******* 2021-02-08T07:09:47.094 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:47.094 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-08T07:09:47.117 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.156 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.068) 0:00:53.814 ******* 2021-02-08T07:09:47.183 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-08T07:09:47.184 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:47.219 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.062) 0:00:53.876 ******* 2021-02-08T07:09:47.225 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:47.226 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-08T07:09:47.248 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.283 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.064) 0:00:53.941 ******* 2021-02-08T07:09:47.322 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-08T07:09:47.323 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.360 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.076) 0:00:54.018 ******* 2021-02-08T07:09:47.366 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:47.367 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-08T07:09:47.389 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-08T07:09:47.390 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.425 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.064) 0:00:54.082 ******* 2021-02-08T07:09:47.454 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-08T07:09:47.456 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi110.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-08T07:09:47.490 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.065) 0:00:54.148 ******* 2021-02-08T07:09:47.518 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-08T07:09:47.519 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.555 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.064) 0:00:54.213 ******* 2021-02-08T07:09:47.584 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:47.585 INFO:teuthology.task.ansible.out:TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi110.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-08T07:09:47.622 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.067) 0:00:54.280 ******* 2021-02-08T07:09:47.650 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-08T07:09:47.650 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:47.689 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.066) 0:00:54.347 ******* 2021-02-08T07:09:47.709 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:47.711 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-08T07:09:47.721 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-08T07:09:47.722 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.757 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.068) 0:00:54.415 ******* 2021-02-08T07:09:47.787 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-08T07:09:47.788 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.827 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.069) 0:00:54.484 ******* 2021-02-08T07:09:47.828 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:47.835 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:47.837 INFO:teuthology.task.ansible.out:TASK [users : Remove revoked users] ******************************************** ok: [smithi110.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-08T07:09:47.849 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:47.850 INFO:teuthology.task.ansible.out:TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.886 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.059) 0:00:54.544 ******* 2021-02-08T07:09:47.926 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-08T07:09:47.928 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:47.965 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:47 +0000 (0:00:00.078) 0:00:54.623 ******* 2021-02-08T07:09:47.971 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-08T07:09:47.972 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-08T07:09:48.026 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.061) 0:00:54.684 ******* 2021-02-08T07:09:48.032 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-08T07:09:48.033 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.071 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.044) 0:00:54.728 ******* 2021-02-08T07:09:48.113 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-08T07:09:48.114 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.147 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.076) 0:00:54.805 ******* 2021-02-08T07:09:48.309 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-08T07:09:48.311 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.011900", "end": "2021-02-08 07:09:48.276344", "rc": 0, "start": "2021-02-08 07:09:48.264444", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-08T07:09:48.366 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.218) 0:00:55.024 ******* 2021-02-08T07:09:48.375 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-08T07:09:48.375 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "checksum": "6bd1cd53e59aef6d9a4a202ec6756843cde9d9e4", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "b908bc6436240ef2dee4e6bf56b67554", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612768188.0468478-158193865399948/source", "state": "file", "uid": 0} 2021-02-08T07:09:48.413 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.046) 0:00:55.070 ******* 2021-02-08T07:09:48.419 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-08T07:09:48.420 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.472 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.059) 0:00:55.130 ******* 2021-02-08T07:09:48.481 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-08T07:09:48.482 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => { "msg": "Host smithi110.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-08T07:09:48.522 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.050) 0:00:55.180 ******* 2021-02-08T07:09:48.529 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-08T07:09:48.529 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.567 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.044) 0:00:55.225 ******* 2021-02-08T07:09:48.574 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-08T07:09:48.575 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-08T07:09:48.615 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.048) 0:00:55.273 ******* 2021-02-08T07:09:48.622 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-08T07:09:48.623 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.623 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:48.625 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.625 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.663 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.047) 0:00:55.320 ******* 2021-02-08T07:09:48.670 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.707 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.044) 0:00:55.365 ******* 2021-02-08T07:09:48.714 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-08T07:09:48.716 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.751 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.043) 0:00:55.408 ******* 2021-02-08T07:09:48.758 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-08T07:09:48.759 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.798 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.045) 0:00:55.454 ******* 2021-02-08T07:09:48.805 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-08T07:09:48.806 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.844 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.047) 0:00:55.501 ******* 2021-02-08T07:09:48.851 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:48.851 INFO:teuthology.task.ansible.out:TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-08T07:09:48.853 INFO:teuthology.task.ansible.out: skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.889 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.045) 0:00:55.547 ******* 2021-02-08T07:09:48.895 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-08T07:09:48.896 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.935 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.045) 0:00:55.593 ******* 2021-02-08T07:09:48.942 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-08T07:09:48.944 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:48.979 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:48 +0000 (0:00:00.044) 0:00:55.637 ******* 2021-02-08T07:09:48.987 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-08T07:09:48.989 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.025 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.045) 0:00:55.683 ******* 2021-02-08T07:09:49.032 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-08T07:09:49.033 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.068 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.042) 0:00:55.726 ******* 2021-02-08T07:09:49.073 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-08T07:09:49.075 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.110 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.042) 0:00:55.768 ******* 2021-02-08T07:09:49.116 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-08T07:09:49.118 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.156 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.044) 0:00:55.813 ******* 2021-02-08T07:09:49.184 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-08T07:09:49.186 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.220 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.064) 0:00:55.878 ******* 2021-02-08T07:09:49.247 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-08T07:09:49.248 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.286 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.065) 0:00:55.944 ******* 2021-02-08T07:09:49.313 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-08T07:09:49.313 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:49.315 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:49.351 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.065) 0:00:56.009 ******* 2021-02-08T07:09:49.378 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-08T07:09:49.379 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.413 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.061) 0:00:56.071 ******* 2021-02-08T07:09:49.439 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.474 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.061) 0:00:56.132 ******* 2021-02-08T07:09:49.501 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-08T07:09:49.501 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:49.539 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.064) 0:00:56.197 ******* 2021-02-08T07:09:49.546 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-08T07:09:49.546 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.296048", "end": "2021-02-08 07:09:49.500399", "rc": 0, "start": "2021-02-08 07:09:49.204351", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:09:49.584 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.044) 0:00:56.241 ******* 2021-02-08T07:09:49.590 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-08T07:09:49.592 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.629 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.045) 0:00:56.287 ******* 2021-02-08T07:09:49.643 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-08T07:09:49.645 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.678 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.049) 0:00:56.336 ******* 2021-02-08T07:09:49.685 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-08T07:09:49.685 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:49.722 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.043) 0:00:56.380 ******* 2021-02-08T07:09:49.748 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-08T07:09:49.750 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.785 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.062) 0:00:56.443 ******* 2021-02-08T07:09:49.811 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-08T07:09:49.813 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.850 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.064) 0:00:56.507 ******* 2021-02-08T07:09:49.855 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-08T07:09:49.855 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-02-08T07:09:49.857 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.041525", "end": "2021-02-08 07:09:49.814705", "rc": 0, "start": "2021-02-08 07:09:49.773180", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:09:49.893 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.043) 0:00:56.551 ******* 2021-02-08T07:09:49.900 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-08T07:09:49.902 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:49.937 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:49 +0000 (0:00:00.044) 0:00:56.595 ******* 2021-02-08T07:09:49.938 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:49.966 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-08T07:09:49.967 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.007 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.070) 0:00:56.665 ******* 2021-02-08T07:09:50.035 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-08T07:09:50.037 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:50.072 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.065) 0:00:56.730 ******* 2021-02-08T07:09:50.099 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-08T07:09:50.100 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.134 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.062) 0:00:56.792 ******* 2021-02-08T07:09:50.161 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-08T07:09:50.162 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.199 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.064) 0:00:56.857 ******* 2021-02-08T07:09:50.225 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-08T07:09:50.226 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.260 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.061) 0:00:56.918 ******* 2021-02-08T07:09:50.287 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-08T07:09:50.289 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.323 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.062) 0:00:56.980 ******* 2021-02-08T07:09:50.349 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-08T07:09:50.350 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.387 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.064) 0:00:57.045 ******* 2021-02-08T07:09:50.413 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-08T07:09:50.415 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.448 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.061) 0:00:57.106 ******* 2021-02-08T07:09:50.474 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-08T07:09:50.476 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:50.511 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.062) 0:00:57.169 ******* 2021-02-08T07:09:50.538 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.573 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.061) 0:00:57.230 ******* 2021-02-08T07:09:50.599 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-08T07:09:50.600 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.634 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.061) 0:00:57.292 ******* 2021-02-08T07:09:50.662 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-08T07:09:50.663 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.698 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.063) 0:00:57.356 ******* 2021-02-08T07:09:50.735 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-08T07:09:50.737 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.773 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.074) 0:00:57.431 ******* 2021-02-08T07:09:50.811 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-08T07:09:50.813 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.851 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.078) 0:00:57.509 ******* 2021-02-08T07:09:50.878 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:50.913 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:50 +0000 (0:00:00.062) 0:00:57.571 ******* 2021-02-08T07:09:51.284 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-08T07:09:51.286 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.247287", "end": "2021-02-08 07:09:51.594572", "rc": 0, "start": "2021-02-08 07:09:51.347285", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-08T07:09:51.320 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:51 +0000 (0:00:00.406) 0:00:57.977 ******* 2021-02-08T07:09:51.347 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-08T07:09:51.349 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.384 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:51 +0000 (0:00:00.064) 0:00:58.042 ******* 2021-02-08T07:09:51.413 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-08T07:09:51.414 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.453 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:51 +0000 (0:00:00.068) 0:00:58.111 ******* 2021-02-08T07:09:51.492 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-08T07:09:51.493 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.503 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.503 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.514 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.555 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:51 +0000 (0:00:00.101) 0:00:58.213 ******* 2021-02-08T07:09:51.582 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-08T07:09:51.584 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.627 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:51 +0000 (0:00:00.072) 0:00:58.285 ******* 2021-02-08T07:09:51.654 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-08T07:09:51.656 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.690 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:51 +0000 (0:00:00.062) 0:00:58.348 ******* 2021-02-08T07:09:51.717 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-08T07:09:51.718 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.754 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:51 +0000 (0:00:00.063) 0:00:58.412 ******* 2021-02-08T07:09:51.781 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-08T07:09:51.782 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.818 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:51 +0000 (0:00:00.064) 0:00:58.476 ******* 2021-02-08T07:09:51.846 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-08T07:09:51.847 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:51.882 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:51 +0000 (0:00:00.063) 0:00:58.540 ******* 2021-02-08T07:09:52.195 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.197202", "end": "2021-02-08 07:09:52.500429", "rc": 0, "start": "2021-02-08 07:09:52.303227", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:09:52.230 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:52 +0000 (0:00:00.348) 0:00:58.888 ******* 2021-02-08T07:09:52.257 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-08T07:09:52.257 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:52.259 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:52.296 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:52 +0000 (0:00:00.066) 0:00:58.954 ******* 2021-02-08T07:09:52.480 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-08T07:09:52.480 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.051596", "end": "2021-02-08 07:09:52.782484", "rc": 0, "start": "2021-02-08 07:09:52.730888", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:09:52.517 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:52 +0000 (0:00:00.220) 0:00:59.175 ******* 2021-02-08T07:09:53.786 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-08T07:09:53.787 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:09:53.821 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:53 +0000 (0:00:01.304) 0:01:00.479 ******* 2021-02-08T07:09:54.722 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-08T07:09:54.723 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:09:54.759 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:54 +0000 (0:00:00.937) 0:01:01.417 ******* 2021-02-08T07:09:54.787 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-08T07:09:54.788 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:54.824 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:54 +0000 (0:00:00.064) 0:01:01.482 ******* 2021-02-08T07:09:55.398 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-08T07:09:55.400 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-02-08T07:09:55.434 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:55 +0000 (0:00:00.610) 0:01:02.092 ******* 2021-02-08T07:09:56.241 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-08T07:09:56.243 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:09:56.281 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:56 +0000 (0:00:00.847) 0:01:02.939 ******* 2021-02-08T07:09:56.340 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-08T07:09:56.342 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:09:56.377 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:56 +0000 (0:00:00.095) 0:01:03.035 ******* 2021-02-08T07:09:56.707 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-08T07:09:56.708 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-02-08T07:09:56.989 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-02-08T07:09:57.114 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-08T07:09:57.116 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:09:57.151 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:57 +0000 (0:00:00.774) 0:01:03.809 ******* 2021-02-08T07:09:57.178 INFO:teuthology.task.ansible.out: 2021-02-08T07:09:57.180 INFO:teuthology.task.ansible.out:TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:57.214 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:57 +0000 (0:00:00.062) 0:01:03.872 ******* 2021-02-08T07:09:57.262 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-08T07:09:57.263 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-02-08T07:09:57.301 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:57 +0000 (0:00:00.086) 0:01:03.958 ******* 2021-02-08T07:09:57.433 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-08T07:09:57.434 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:09:57.473 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:57 +0000 (0:00:00.172) 0:01:04.131 ******* 2021-02-08T07:09:57.531 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-08T07:09:57.532 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-02-08T07:09:57.576 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:57 +0000 (0:00:00.102) 0:01:04.233 ******* 2021-02-08T07:09:57.780 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-08T07:09:57.782 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-02-08T07:09:58.033 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-02-08T07:09:58.304 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-02-08T07:09:58.348 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:58 +0000 (0:00:00.772) 0:01:05.006 ******* 2021-02-08T07:09:58.386 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-08T07:09:58.387 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-02-08T07:09:58.422 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:58 +0000 (0:00:00.073) 0:01:05.080 ******* 2021-02-08T07:09:58.429 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-08T07:09:58.429 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:09:58.467 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:58 +0000 (0:00:00.044) 0:01:05.125 ******* 2021-02-08T07:09:58.474 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-08T07:09:58.476 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:58.514 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:58 +0000 (0:00:00.046) 0:01:05.172 ******* 2021-02-08T07:09:58.542 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-08T07:09:58.544 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:09:58.581 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:58 +0000 (0:00:00.067) 0:01:05.239 ******* 2021-02-08T07:09:58.620 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-08T07:09:58.622 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:58.631 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:58.643 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:58.644 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-08T07:09:58.687 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:58 +0000 (0:00:00.105) 0:01:05.344 ******* 2021-02-08T07:09:58.726 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-08T07:09:58.727 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:09:58.764 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:58 +0000 (0:00:00.077) 0:01:05.422 ******* 2021-02-08T07:09:58.886 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-08T07:09:58.888 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-02-08T07:09:59.225 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-02-08T07:09:59.542 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-02-08T07:09:59.580 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:59 +0000 (0:00:00.816) 0:01:06.238 ******* 2021-02-08T07:09:59.713 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:09:59.750 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:59 +0000 (0:00:00.170) 0:01:06.408 ******* 2021-02-08T07:09:59.758 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-08T07:09:59.759 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:09:59.812 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:09:59 +0000 (0:00:00.061) 0:01:06.470 ******* 2021-02-08T07:10:00.098 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-08T07:10:00.099 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-02-08T07:10:00.138 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:00 +0000 (0:00:00.325) 0:01:06.796 ******* 2021-02-08T07:10:00.180 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-08T07:10:00.181 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-02-08T07:10:00.481 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:00.483 INFO:teuthology.task.ansible.out:TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-02-08T07:10:00.484 INFO:teuthology.task.ansible.out:TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi110.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-02-08T07:10:00.532 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:00 +0000 (0:00:00.394) 0:01:07.190 ******* 2021-02-08T07:10:00.681 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-08T07:10:00.682 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-02-08T07:10:00.798 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-08T07:10:00.799 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-02-08T07:10:00.800 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-08T07:10:00.801 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-02-08T07:10:00.838 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:00 +0000 (0:00:00.306) 0:01:07.496 ******* 2021-02-08T07:10:00.878 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:00 +0000 (0:00:00.040) 0:01:07.536 ******* 2021-02-08T07:10:00.885 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-08T07:10:00.887 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:00.925 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:00 +0000 (0:00:00.046) 0:01:07.583 ******* 2021-02-08T07:10:00.932 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-08T07:10:00.934 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-02-08T07:10:00.968 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:00 +0000 (0:00:00.043) 0:01:07.626 ******* 2021-02-08T07:10:00.975 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-08T07:10:00.976 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:10:01.012 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:01 +0000 (0:00:00.043) 0:01:07.670 ******* 2021-02-08T07:10:01.019 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-08T07:10:01.020 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:01.056 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:01 +0000 (0:00:00.044) 0:01:07.714 ******* 2021-02-08T07:10:01.062 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-08T07:10:01.063 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:01.103 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:01 +0000 (0:00:00.046) 0:01:07.761 ******* 2021-02-08T07:10:01.109 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-08T07:10:01.111 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:10:01.148 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:01 +0000 (0:00:00.044) 0:01:07.806 ******* 2021-02-08T07:10:01.185 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-08T07:10:01.187 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:01.197 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:01.208 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:01.209 INFO:teuthology.task.ansible.out: skipping: [smithi110.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:01.256 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:01 +0000 (0:00:00.108) 0:01:07.914 ******* 2021-02-08T07:10:01.284 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:01.328 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:01 +0000 (0:00:00.072) 0:01:07.986 ******* 2021-02-08T07:10:01.430 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-08T07:10:01.431 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-02-08T07:10:01.471 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:01 +0000 (0:00:00.142) 0:01:08.128 ******* 2021-02-08T07:10:02.154 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-08T07:10:02.155 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-02-08 07:08:21 UTC", "ActiveEnterTimestampMonotonic": "19065769", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount var-run.mount local-fs.target basic.target nss-lookup.target systemd-journald.socket sysinit.target remote-fs.target systemd-tmpfiles-setup.service network.target system.slice time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-02-08 07:08:21 UTC", "AssertTimestampMonotonic": "19064615", "Before": "getty@tty1.service xdm.service plymouth-quit.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-02-08 07:08:21 UTC", "ConditionTimestampMonotonic": "19064615", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-02-08T07:10:02.156 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1277", "ExecMainStartTimestamp": "Mon 2021-02-08 07:08:21 UTC", "ExecMainStartTimestampMonotonic": "19065721", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Mon 2021-02-08 07:08:21 UTC] ; stop_time=[n/a] ; pid=1277 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-02-08 07:08:21 UTC", "InactiveExitTimestampMonotonic": "19065769", "InvocationID": "bab940f323884061b43bc885adecae6a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-02-08T07:10:02.158 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1277", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-02-08T07:10:02.159 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-02-08 07:08:21 UTC", "StateChangeTimestampMonotonic": "19065769", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-02-08 07:08:21 UTC", "WatchdogTimestampMonotonic": "19065768", "WatchdogUSec": "0"}} 2021-02-08T07:10:02.193 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.722) 0:01:08.851 ******* 2021-02-08T07:10:02.231 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-08T07:10:02.232 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:10:02.269 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.076) 0:01:08.927 ******* 2021-02-08T07:10:02.303 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.033) 0:01:08.961 ******* 2021-02-08T07:10:02.309 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-08T07:10:02.310 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:02.347 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.043) 0:01:09.005 ******* 2021-02-08T07:10:02.373 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-08T07:10:02.374 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:02.408 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.061) 0:01:09.066 ******* 2021-02-08T07:10:02.436 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-08T07:10:02.438 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:02.476 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.068) 0:01:09.134 ******* 2021-02-08T07:10:02.504 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-08T07:10:02.506 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:02.543 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.066) 0:01:09.201 ******* 2021-02-08T07:10:02.574 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-08T07:10:02.576 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:02.617 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.073) 0:01:09.275 ******* 2021-02-08T07:10:02.624 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-08T07:10:02.624 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-02-08T07:10:02.661 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.043) 0:01:09.318 ******* 2021-02-08T07:10:02.667 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-08T07:10:02.668 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:02.706 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.045) 0:01:09.364 ******* 2021-02-08T07:10:02.734 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-08T07:10:02.735 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:02.771 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.064) 0:01:09.429 ******* 2021-02-08T07:10:02.811 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-08T07:10:02.812 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:02.859 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.086) 0:01:09.516 ******* 2021-02-08T07:10:02.887 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-08T07:10:02.889 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:02.922 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.064) 0:01:09.580 ******* 2021-02-08T07:10:02.949 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-08T07:10:02.951 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-02-08T07:10:02.991 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:02 +0000 (0:00:00.068) 0:01:09.649 ******* 2021-02-08T07:10:02.999 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-08T07:10:02.999 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-02-08T07:10:03.038 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.047) 0:01:09.696 ******* 2021-02-08T07:10:03.045 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-08T07:10:03.045 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-02-08T07:10:03.085 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.046) 0:01:09.743 ******* 2021-02-08T07:10:03.124 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-08T07:10:03.125 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-02-08T07:10:03.164 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.078) 0:01:09.821 ******* 2021-02-08T07:10:03.191 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-08T07:10:03.192 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-02-08T07:10:03.204 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:03.206 INFO:teuthology.task.ansible.out:TASK [testnode : Including version specific variables.] ************************ ok: [smithi017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-08T07:10:03.243 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.079) 0:01:09.901 ******* 2021-02-08T07:10:03.304 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-08T07:10:03.305 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-02-08T07:10:03.360 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.116) 0:01:10.018 ******* 2021-02-08T07:10:03.377 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-08T07:10:03.379 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-08T07:10:03.417 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.057) 0:01:10.075 ******* 2021-02-08T07:10:03.424 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-08T07:10:03.425 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:03.464 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.046) 0:01:10.122 ******* 2021-02-08T07:10:03.514 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-08T07:10:03.515 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-08T07:10:03.558 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.094) 0:01:10.216 ******* 2021-02-08T07:10:03.564 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-08T07:10:03.565 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-02-08T07:10:03.604 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.045) 0:01:10.262 ******* 2021-02-08T07:10:03.759 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-08T07:10:03.761 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:03.796 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.192) 0:01:10.454 ******* 2021-02-08T07:10:03.802 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-08T07:10:03.806 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:03.844 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.047) 0:01:10.502 ******* 2021-02-08T07:10:03.882 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-08T07:10:03.883 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:03.918 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:03 +0000 (0:00:00.073) 0:01:10.576 ******* 2021-02-08T07:10:04.071 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-08T07:10:04.073 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-02-08T07:10:04.111 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.192) 0:01:10.769 ******* 2021-02-08T07:10:04.119 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-08T07:10:04.119 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-02-08T07:10:04.158 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.047) 0:01:10.816 ******* 2021-02-08T07:10:04.386 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-08T07:10:04.387 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-02-08 07:08:36 UTC", "ActiveEnterTimestampMonotonic": "21423313", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice systemd-tmpfiles-setup.service sysinit.target systemd-journald.socket -.mount var-run.mount remote-fs.target local-fs.target network.target time-sync.target nss-lookup.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-02-08 07:08:36 UTC", "AssertTimestampMonotonic": "21422066", "Before": "xdm.service shutdown.target getty@tty1.service multi-user.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-02-08 07:08:36 UTC", "ConditionTimestampMonotonic": "21422066", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-02-08T07:10:04.389 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1281", "ExecMainStartTimestamp": "Mon 2021-02-08 07:08:36 UTC", "ExecMainStartTimestampMonotonic": "21423272", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Mon 2021-02-08 07:08:36 UTC] ; stop_time=[n/a] ; pid=1281 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-02-08 07:08:36 UTC", "InactiveExitTimestampMonotonic": "21423313", "InvocationID": "ae96a07e393c47019f01690fc3c0de7d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-02-08T07:10:04.390 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128028", "LimitNPROCSoft": "128028", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "128028", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1281", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-02-08T07:10:04.391 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-02-08 07:08:36 UTC", "StateChangeTimestampMonotonic": "21423313", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-02-08 07:08:36 UTC", "WatchdogTimestampMonotonic": "21423312", "WatchdogUSec": "0"}} 2021-02-08T07:10:04.423 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.265) 0:01:11.081 ******* 2021-02-08T07:10:04.450 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-08T07:10:04.451 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:04.486 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.062) 0:01:11.144 ******* 2021-02-08T07:10:04.512 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-08T07:10:04.514 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:04.549 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.063) 0:01:11.207 ******* 2021-02-08T07:10:04.583 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.033) 0:01:11.240 ******* 2021-02-08T07:10:04.588 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-08T07:10:04.588 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi017"}, "changed": false} 2021-02-08T07:10:04.649 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.066) 0:01:11.307 ******* 2021-02-08T07:10:04.661 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-08T07:10:04.662 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:04.698 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.049) 0:01:11.356 ******* 2021-02-08T07:10:04.736 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:04.776 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.076) 0:01:11.433 ******* 2021-02-08T07:10:04.803 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-08T07:10:04.805 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:04.851 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.075) 0:01:11.508 ******* 2021-02-08T07:10:04.881 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-08T07:10:04.882 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:04.923 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.072) 0:01:11.580 ******* 2021-02-08T07:10:04.949 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-08T07:10:04.951 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:04.985 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:04 +0000 (0:00:00.062) 0:01:11.643 ******* 2021-02-08T07:10:05.011 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-08T07:10:05.013 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:05.061 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.075) 0:01:11.718 ******* 2021-02-08T07:10:05.089 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-08T07:10:05.091 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:05.137 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.076) 0:01:11.794 ******* 2021-02-08T07:10:05.175 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-08T07:10:05.177 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:05.212 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.075) 0:01:11.870 ******* 2021-02-08T07:10:05.249 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-08T07:10:05.250 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-02-08T07:10:05.285 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.073) 0:01:11.943 ******* 2021-02-08T07:10:05.344 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-08T07:10:05.345 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-02-08T07:10:05.382 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.096) 0:01:12.039 ******* 2021-02-08T07:10:05.419 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi110.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-02-08T07:10:05.456 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.074) 0:01:12.114 ******* 2021-02-08T07:10:05.462 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-08T07:10:05.463 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi017.front.sepia.ceph.com", "ansible_hostname": "smithi017", "ansible_nodename": "smithi017"}, "changed": true, "name": "smithi017"} 2021-02-08T07:10:05.509 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.052) 0:01:12.167 ******* 2021-02-08T07:10:05.516 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-08T07:10:05.518 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-08T07:10:05.556 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.046) 0:01:12.214 ******* 2021-02-08T07:10:05.689 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-08T07:10:05.690 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-08T07:10:05.737 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.181) 0:01:12.395 ******* 2021-02-08T07:10:05.841 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-08T07:10:05.842 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-02-08T07:10:05.882 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.144) 0:01:12.540 ******* 2021-02-08T07:10:05.889 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-08T07:10:05.889 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-02-08T07:10:05.930 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:05 +0000 (0:00:00.048) 0:01:12.588 ******* 2021-02-08T07:10:05.968 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-08T07:10:05.970 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:06.004 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:06 +0000 (0:00:00.073) 0:01:12.661 ******* 2021-02-08T07:10:06.032 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:06.070 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:06 +0000 (0:00:00.066) 0:01:12.728 ******* 2021-02-08T07:10:06.198 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-08T07:10:06.199 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-08T07:10:06.239 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:06 +0000 (0:00:00.169) 0:01:12.897 ******* 2021-02-08T07:10:06.247 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-08T07:10:06.247 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-02-08T07:10:06.284 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:06 +0000 (0:00:00.044) 0:01:12.942 ******* 2021-02-08T07:10:06.671 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:06.673 INFO:teuthology.task.ansible.out:TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-02-08T07:10:06.708 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:06 +0000 (0:00:00.423) 0:01:13.365 ******* 2021-02-08T07:10:06.745 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi110"}, "changed": false} 2021-02-08T07:10:06.798 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:06 +0000 (0:00:00.090) 0:01:13.456 ******* 2021-02-08T07:10:06.878 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-08T07:10:06.879 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-08T07:10:06.917 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:06 +0000 (0:00:00.119) 0:01:13.575 ******* 2021-02-08T07:10:06.945 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-08T07:10:06.946 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:06.980 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:06 +0000 (0:00:00.062) 0:01:13.638 ******* 2021-02-08T07:10:07.007 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-08T07:10:07.007 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.047 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.066) 0:01:13.705 ******* 2021-02-08T07:10:07.074 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:07.076 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.112 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.064) 0:01:13.770 ******* 2021-02-08T07:10:07.139 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-08T07:10:07.140 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.176 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.064) 0:01:13.834 ******* 2021-02-08T07:10:07.203 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-08T07:10:07.205 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.240 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.063) 0:01:13.897 ******* 2021-02-08T07:10:07.268 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-08T07:10:07.269 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.303 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.063) 0:01:13.961 ******* 2021-02-08T07:10:07.330 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-08T07:10:07.331 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.365 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.062) 0:01:14.023 ******* 2021-02-08T07:10:07.372 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-08T07:10:07.373 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi110.front.sepia.ceph.com", "ansible_hostname": "smithi110", "ansible_nodename": "smithi110"}, "changed": true, "name": "smithi110"} 2021-02-08T07:10:07.409 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.043) 0:01:14.067 ******* 2021-02-08T07:10:07.416 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-08T07:10:07.418 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.454 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.045) 0:01:14.112 ******* 2021-02-08T07:10:07.481 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-08T07:10:07.518 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.063) 0:01:14.176 ******* 2021-02-08T07:10:07.544 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-08T07:10:07.583 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.065) 0:01:14.241 ******* 2021-02-08T07:10:07.610 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-08T07:10:07.646 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.063) 0:01:14.304 ******* 2021-02-08T07:10:07.680 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-08T07:10:07.681 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.715 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.069) 0:01:14.373 ******* 2021-02-08T07:10:07.732 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-08T07:10:07.734 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-02-08T07:10:07.770 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.055) 0:01:14.428 ******* 2021-02-08T07:10:07.776 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-08T07:10:07.778 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.824 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.053) 0:01:14.482 ******* 2021-02-08T07:10:07.851 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-08T07:10:07.853 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.886 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.062) 0:01:14.544 ******* 2021-02-08T07:10:07.916 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:07.917 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:07.950 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:07 +0000 (0:00:00.063) 0:01:14.608 ******* 2021-02-08T07:10:07.978 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-08T07:10:07.980 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.013 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.062) 0:01:14.671 ******* 2021-02-08T07:10:08.041 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-08T07:10:08.043 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.078 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.064) 0:01:14.735 ******* 2021-02-08T07:10:08.096 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-08T07:10:08.097 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-08T07:10:08.132 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.054) 0:01:14.790 ******* 2021-02-08T07:10:08.140 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-08T07:10:08.140 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.178 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.045) 0:01:14.836 ******* 2021-02-08T07:10:08.206 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.240 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.062) 0:01:14.898 ******* 2021-02-08T07:10:08.267 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-08T07:10:08.269 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.303 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.062) 0:01:14.961 ******* 2021-02-08T07:10:08.331 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-08T07:10:08.333 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.369 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.065) 0:01:15.027 ******* 2021-02-08T07:10:08.397 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-08T07:10:08.398 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.432 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.062) 0:01:15.090 ******* 2021-02-08T07:10:08.459 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:08.460 INFO:teuthology.task.ansible.out:TASK [testnode : Install epel packages] **************************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.494 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.062) 0:01:15.152 ******* 2021-02-08T07:10:08.523 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-08T07:10:08.524 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.559 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.064) 0:01:15.217 ******* 2021-02-08T07:10:08.591 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-08T07:10:08.592 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.628 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.069) 0:01:15.286 ******* 2021-02-08T07:10:08.656 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-08T07:10:08.658 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.694 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.065) 0:01:15.352 ******* 2021-02-08T07:10:08.727 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.033) 0:01:15.385 ******* 2021-02-08T07:10:08.734 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-08T07:10:08.736 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.770 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.042) 0:01:15.428 ******* 2021-02-08T07:10:08.777 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-08T07:10:08.778 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.814 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.043) 0:01:15.472 ******* 2021-02-08T07:10:08.821 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.856 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.042) 0:01:15.514 ******* 2021-02-08T07:10:08.862 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-08T07:10:08.864 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.900 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.044) 0:01:15.558 ******* 2021-02-08T07:10:08.907 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-08T07:10:08.908 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.943 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.043) 0:01:15.601 ******* 2021-02-08T07:10:08.950 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-08T07:10:08.951 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:08.988 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:08 +0000 (0:00:00.044) 0:01:15.646 ******* 2021-02-08T07:10:08.994 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-08T07:10:08.995 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.033 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.045) 0:01:15.691 ******* 2021-02-08T07:10:09.035 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:09.043 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:09.044 INFO:teuthology.task.ansible.out:TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.080 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.046) 0:01:15.738 ******* 2021-02-08T07:10:09.087 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-08T07:10:09.089 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.124 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.043) 0:01:15.782 ******* 2021-02-08T07:10:09.130 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-08T07:10:09.130 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.166 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.042) 0:01:15.824 ******* 2021-02-08T07:10:09.172 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-08T07:10:09.179 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.214 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.046) 0:01:15.871 ******* 2021-02-08T07:10:09.220 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-08T07:10:09.221 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.257 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.043) 0:01:15.915 ******* 2021-02-08T07:10:09.264 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-08T07:10:09.266 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.301 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.043) 0:01:15.958 ******* 2021-02-08T07:10:09.308 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-08T07:10:09.309 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.346 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.045) 0:01:16.004 ******* 2021-02-08T07:10:09.372 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-08T07:10:09.374 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.408 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.062) 0:01:16.066 ******* 2021-02-08T07:10:09.436 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-08T07:10:09.437 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005055", "end": "2021-02-08 07:10:09.418867", "rc": 0, "start": "2021-02-08 07:10:09.413812", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:09.472 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.063) 0:01:16.130 ******* 2021-02-08T07:10:09.507 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.034) 0:01:16.164 ******* 2021-02-08T07:10:09.513 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-08T07:10:09.550 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.043) 0:01:16.208 ******* 2021-02-08T07:10:09.557 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-08T07:10:09.595 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.044) 0:01:16.253 ******* 2021-02-08T07:10:09.622 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-08T07:10:09.660 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.065) 0:01:16.318 ******* 2021-02-08T07:10:09.688 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.724 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.063) 0:01:16.382 ******* 2021-02-08T07:10:09.762 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-08T07:10:09.764 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.799 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.074) 0:01:16.456 ******* 2021-02-08T07:10:09.826 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-08T07:10:09.827 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.862 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.062) 0:01:16.519 ******* 2021-02-08T07:10:09.878 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-08T07:10:09.880 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-02-08T07:10:09.918 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.056) 0:01:16.576 ******* 2021-02-08T07:10:09.924 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:09.926 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:09.962 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:09 +0000 (0:00:00.044) 0:01:16.620 ******* 2021-02-08T07:10:09.972 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-08T07:10:09.973 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.009 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.046) 0:01:16.667 ******* 2021-02-08T07:10:10.016 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-08T07:10:10.018 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.053 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.044) 0:01:16.711 ******* 2021-02-08T07:10:10.080 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.116 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.062) 0:01:16.774 ******* 2021-02-08T07:10:10.145 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-08T07:10:10.146 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.182 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.066) 0:01:16.840 ******* 2021-02-08T07:10:10.209 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-08T07:10:10.210 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.246 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.064) 0:01:16.904 ******* 2021-02-08T07:10:10.273 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-08T07:10:10.274 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.309 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.062) 0:01:16.967 ******* 2021-02-08T07:10:10.336 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.371 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.062) 0:01:17.029 ******* 2021-02-08T07:10:10.398 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-08T07:10:10.400 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.436 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.064) 0:01:17.094 ******* 2021-02-08T07:10:10.463 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-08T07:10:10.464 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.499 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.062) 0:01:17.156 ******* 2021-02-08T07:10:10.525 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-08T07:10:10.527 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.562 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.063) 0:01:17.220 ******* 2021-02-08T07:10:10.589 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-08T07:10:10.591 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.624 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.062) 0:01:17.282 ******* 2021-02-08T07:10:10.651 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-08T07:10:10.652 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.690 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.065) 0:01:17.348 ******* 2021-02-08T07:10:10.717 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-08T07:10:10.718 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.754 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.063) 0:01:17.412 ******* 2021-02-08T07:10:10.781 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-08T07:10:10.782 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.818 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.063) 0:01:17.476 ******* 2021-02-08T07:10:10.844 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-08T07:10:10.846 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.881 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.062) 0:01:17.538 ******* 2021-02-08T07:10:10.907 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:10.908 INFO:teuthology.task.ansible.out:TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:10.943 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:10 +0000 (0:00:00.062) 0:01:17.601 ******* 2021-02-08T07:10:10.970 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-08T07:10:10.971 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:11.006 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:11 +0000 (0:00:00.062) 0:01:17.664 ******* 2021-02-08T07:10:11.033 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-08T07:10:11.034 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:11.069 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:11 +0000 (0:00:00.063) 0:01:17.727 ******* 2021-02-08T07:10:11.098 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-08T07:10:11.100 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:11.136 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:11 +0000 (0:00:00.067) 0:01:17.794 ******* 2021-02-08T07:10:11.279 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-08T07:10:11.281 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} TASK [testnode : Check for custom repos] *************************************** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005222", "end": "2021-02-08 07:10:11.581202", "rc": 0, "start": "2021-02-08 07:10:11.575980", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:11.320 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:11 +0000 (0:00:00.183) 0:01:17.977 ******* 2021-02-08T07:10:11.347 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-08T07:10:11.386 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:11 +0000 (0:00:00.066) 0:01:18.044 ******* 2021-02-08T07:10:11.720 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-08T07:10:11.721 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-02-08T07:10:11.758 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:11 +0000 (0:00:00.371) 0:01:18.416 ******* 2021-02-08T07:10:11.795 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-08T07:10:11.797 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:11.835 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:11 +0000 (0:00:00.076) 0:01:18.493 ******* 2021-02-08T07:10:12.249 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-08T07:10:12.251 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-02-08T07:10:12.287 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:12 +0000 (0:00:00.451) 0:01:18.944 ******* 2021-02-08T07:10:12.891 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-02-08T07:10:13.215 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-08T07:10:13.217 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:10:13.251 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:13 +0000 (0:00:00.964) 0:01:19.909 ******* 2021-02-08T07:10:13.289 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-08T07:10:13.325 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:13 +0000 (0:00:00.074) 0:01:19.983 ******* 2021-02-08T07:10:13.498 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-08T07:10:13.500 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036586", "end": "2021-02-08 07:10:13.476249", "rc": 0, "start": "2021-02-08 07:10:13.439663", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:13.536 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:13 +0000 (0:00:00.210) 0:01:20.193 ******* 2021-02-08T07:10:13.825 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-08T07:10:13.826 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-02-08T07:10:13.863 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:13 +0000 (0:00:00.327) 0:01:20.521 ******* 2021-02-08T07:10:14.888 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-08T07:10:14.889 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:10:14.923 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:14 +0000 (0:00:01.060) 0:01:21.581 ******* 2021-02-08T07:10:14.952 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:14.953 INFO:teuthology.task.ansible.out:TASK [testnode : Add local apt repos.] ***************************************** 2021-02-08T07:10:14.994 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:14 +0000 (0:00:00.070) 0:01:21.652 ******* 2021-02-08T07:10:15.172 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-08T07:10:15.174 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.033919", "end": "2021-02-08 07:10:15.464386", "rc": 0, "start": "2021-02-08 07:10:15.430467", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:15.213 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:15 +0000 (0:00:00.218) 0:01:21.871 ******* 2021-02-08T07:10:16.586 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-08T07:10:16.587 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:10:16.622 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:16 +0000 (0:00:01.409) 0:01:23.280 ******* 2021-02-08T07:10:18.298 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:10:18.336 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:18 +0000 (0:00:01.713) 0:01:24.994 ******* 2021-02-08T07:10:25.734 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-08T07:10:25.735 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 11 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directorie 2021-02-08T07:10:25.736 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 11 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-02-08T07:10:25.737 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-08T07:10:25.773 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:25 +0000 (0:00:07.436) 0:01:32.430 ******* 2021-02-08T07:10:26.569 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 11 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directorie 2021-02-08T07:10:26.570 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 11 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-02-08T07:10:26.571 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-08T07:10:26.606 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:26 +0000 (0:00:00.833) 0:01:33.264 ******* 2021-02-08T07:10:26.718 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-08T07:10:26.719 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:10:26.756 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:26 +0000 (0:00:00.149) 0:01:33.414 ******* 2021-02-08T07:10:26.784 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:26.821 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:26 +0000 (0:00:00.065) 0:01:33.479 ******* 2021-02-08T07:10:27.532 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-08T07:10:27.534 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:10:27.569 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:27 +0000 (0:00:00.747) 0:01:34.226 ******* 2021-02-08T07:10:27.605 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-08T07:10:27.607 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:27.641 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:27 +0000 (0:00:00.072) 0:01:34.299 ******* 2021-02-08T07:10:35.795 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-08T07:10:35.931 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 11 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.34 [40.2 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.34 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.34 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.34 [871 kB]\nFetched 10.4 MB in 1s (9401 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-02-08T07:10:35.932 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-02-08T07:10:35.934 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 11 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.34 [40.2 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.34 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.34 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.34 [871 kB]", "Fetched 10.4 MB in 1s (9401 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-02-08T07:10:35.935 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} Monday 08 February 2021 07:10:35 +0000 (0:00:08.194) 0:01:42.493 ******* 2021-02-08T07:10:36.098 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 11 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.34 [40.2 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.34 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.34 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.34 [871 kB]\nFetched 10.4 MB in 1s (9808 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-02-08T07:10:36.099 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-02-08T07:10:36.101 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 11 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.34 [40.2 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.34 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.34 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.34 [871 kB]", "Fetched 10.4 MB in 1s (9808 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-02-08T07:10:36.102 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-08T07:10:36.140 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:36 +0000 (0:00:00.304) 0:01:42.798 ******* 2021-02-08T07:10:36.760 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-08T07:10:36.762 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:10:36.800 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:36 +0000 (0:00:00.659) 0:01:43.457 ******* 2021-02-08T07:10:36.933 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-08T07:10:36.934 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-08T07:10:36.969 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:36 +0000 (0:00:00.169) 0:01:43.627 ******* 2021-02-08T07:10:36.996 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-08T07:10:36.997 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:37.032 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.062) 0:01:43.690 ******* 2021-02-08T07:10:37.101 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-08T07:10:37.103 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:10:37.139 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.107) 0:01:43.797 ******* 2021-02-08T07:10:37.170 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-08T07:10:37.171 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-02-08T07:10:37.207 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.068) 0:01:43.865 ******* 2021-02-08T07:10:37.278 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-08T07:10:37.314 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.106) 0:01:43.972 ******* 2021-02-08T07:10:37.354 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:37.387 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.072) 0:01:44.045 ******* 2021-02-08T07:10:37.520 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-08T07:10:37.522 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-08T07:10:37.523 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-02-08T07:10:37.559 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.172) 0:01:44.217 ******* 2021-02-08T07:10:37.596 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.036) 0:01:44.254 ******* 2021-02-08T07:10:37.711 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi017.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-08T07:10:37.748 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.151) 0:01:44.406 ******* 2021-02-08T07:10:37.881 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:37.883 INFO:teuthology.task.ansible.out:TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-08T07:10:37.884 INFO:teuthology.task.ansible.out:TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-08T07:10:37.924 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.175) 0:01:44.582 ******* 2021-02-08T07:10:37.957 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:37 +0000 (0:00:00.033) 0:01:44.615 ******* 2021-02-08T07:10:38.120 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-08T07:10:38.121 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-08T07:10:38.156 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.199) 0:01:44.814 ******* 2021-02-08T07:10:38.204 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-08T07:10:38.206 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2021-02-08 07:08:22 UTC", "ActiveEnterTimestampMonotonic": "19566942", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target remote-fs.target -.mount basic.target systemd-journald.socket network.target systemd-tmpfiles-setup.service system.slice nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-02-08 07:08:21 UTC", "AssertTimestampMonotonic": "19066788", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-02-08 07:08:21 UTC", "ConditionTimestampMonotonic": "19066788", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-02-08T07:10:38.207 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1377", "ExecMainStartTimestamp": "Mon 2021-02-08 07:08:22 UTC", "ExecMainStartTimestampMonotonic": "19566923", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-02-08 07:08:21 UTC", "InactiveExitTimestampMonotonic": "19067723", "InvocationID": "106c3b2fdec745bab1acf77d6470d27d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-02-08T07:10:38.208 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1377", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-02-08T07:10:38.210 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-02-08 07:08:22 UTC", "StateChangeTimestampMonotonic": "19566942", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-02-08 07:08:22 UTC", "WatchdogTimestampMonotonic": "19566941", "WatchdogUSec": "0"}} 2021-02-08T07:10:38.244 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.087) 0:01:44.902 ******* 2021-02-08T07:10:38.272 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-08T07:10:38.274 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.309 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.064) 0:01:44.967 ******* 2021-02-08T07:10:38.315 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-08T07:10:38.316 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-08T07:10:38.352 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.042) 0:01:45.010 ******* 2021-02-08T07:10:38.359 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-08T07:10:38.360 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.400 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.048) 0:01:45.058 ******* 2021-02-08T07:10:38.429 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.464 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.063) 0:01:45.122 ******* 2021-02-08T07:10:38.491 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-08T07:10:38.492 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.528 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.063) 0:01:45.186 ******* 2021-02-08T07:10:38.556 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-08T07:10:38.558 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.593 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.065) 0:01:45.251 ******* 2021-02-08T07:10:38.594 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:38.621 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:38.622 INFO:teuthology.task.ansible.out:TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.659 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.066) 0:01:45.317 ******* 2021-02-08T07:10:38.668 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2021-02-08 07:08:36 UTC", "ActiveEnterTimestampMonotonic": "21799845", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target nss-lookup.target network.target -.mount systemd-tmpfiles-setup.service remote-fs.target system.slice systemd-journald.socket basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-02-08 07:08:36 UTC", "AssertTimestampMonotonic": "21420834", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-02-08 07:08:36 UTC", "ConditionTimestampMonotonic": "21420834", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", 2021-02-08T07:10:38.670 INFO:teuthology.task.ansible.out:"Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1342", "ExecMainStartTimestamp": "Mon 2021-02-08 07:08:36 UTC", "ExecMainStartTimestampMonotonic": "21799827", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-02-08 07:08:36 UTC", "InactiveExitTimestampMonotonic": "21421445", "InvocationID": "101b29aa6869418bb4b915578643913d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA 2021-02-08T07:10:38.671 INFO:teuthology.task.ansible.out:": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128028", "LimitNPROCSoft": "128028", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "128028", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1342", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", 2021-02-08T07:10:38.672 INFO:teuthology.task.ansible.out: "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-02-08 07:08:36 UTC", "StateChangeTimestampMonotonic": "21799845", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-02-08 07:08:36 UTC", "WatchdogTimestampMonotonic": "21799843", "WatchdogUSec": "0"}} 2021-02-08T07:10:38.704 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.044) 0:01:45.362 ******* 2021-02-08T07:10:38.710 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-08T07:10:38.711 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.749 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.045) 0:01:45.407 ******* 2021-02-08T07:10:38.755 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-08T07:10:38.757 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.795 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.045) 0:01:45.452 ******* 2021-02-08T07:10:38.801 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-08T07:10:38.801 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.837 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.042) 0:01:45.495 ******* 2021-02-08T07:10:38.845 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-08T07:10:38.846 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.881 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.043) 0:01:45.539 ******* 2021-02-08T07:10:38.887 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:38.888 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.924 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.043) 0:01:45.582 ******* 2021-02-08T07:10:38.931 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-08T07:10:38.932 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:38.969 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:38 +0000 (0:00:00.044) 0:01:45.627 ******* 2021-02-08T07:10:38.975 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-08T07:10:38.976 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.013 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.044) 0:01:45.671 ******* 2021-02-08T07:10:39.020 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-08T07:10:39.020 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.059 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.045) 0:01:45.716 ******* 2021-02-08T07:10:39.065 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-08T07:10:39.066 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.101 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.042) 0:01:45.759 ******* 2021-02-08T07:10:39.108 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-08T07:10:39.111 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.144 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.042) 0:01:45.802 ******* 2021-02-08T07:10:39.151 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-08T07:10:39.152 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.187 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.042) 0:01:45.845 ******* 2021-02-08T07:10:39.194 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-08T07:10:39.195 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.250 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.062) 0:01:45.908 ******* 2021-02-08T07:10:39.258 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-08T07:10:39.259 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.299 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.049) 0:01:45.957 ******* 2021-02-08T07:10:39.308 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-08T07:10:39.310 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.349 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.050) 0:01:46.007 ******* 2021-02-08T07:10:39.357 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-08T07:10:39.358 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.397 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.047) 0:01:46.055 ******* 2021-02-08T07:10:39.404 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-08T07:10:39.404 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.446 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.048) 0:01:46.104 ******* 2021-02-08T07:10:39.452 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-08T07:10:39.452 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.490 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.044) 0:01:46.148 ******* 2021-02-08T07:10:39.498 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:39.499 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.540 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.050) 0:01:46.198 ******* 2021-02-08T07:10:39.548 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-08T07:10:39.550 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.591 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.050) 0:01:46.249 ******* 2021-02-08T07:10:39.598 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-08T07:10:39.600 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.640 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.047) 0:01:46.296 ******* 2021-02-08T07:10:39.646 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-08T07:10:39.648 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.695 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.056) 0:01:46.353 ******* 2021-02-08T07:10:39.702 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-08T07:10:39.703 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.757 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.061) 0:01:46.415 ******* 2021-02-08T07:10:39.763 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-08T07:10:39.764 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.803 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.046) 0:01:46.461 ******* 2021-02-08T07:10:39.813 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.854 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.050) 0:01:46.512 ******* 2021-02-08T07:10:39.861 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-08T07:10:39.863 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.898 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.044) 0:01:46.556 ******* 2021-02-08T07:10:39.905 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-08T07:10:39.907 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.949 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.050) 0:01:46.607 ******* 2021-02-08T07:10:39.955 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-08T07:10:39.957 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:39.996 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:39 +0000 (0:00:00.047) 0:01:46.654 ******* 2021-02-08T07:10:40.002 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-08T07:10:40.003 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.040 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.044) 0:01:46.698 ******* 2021-02-08T07:10:40.046 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-08T07:10:40.048 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.085 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.044) 0:01:46.743 ******* 2021-02-08T07:10:40.091 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-08T07:10:40.092 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.135 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.050) 0:01:46.793 ******* 2021-02-08T07:10:40.141 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-08T07:10:40.142 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.198 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.063) 0:01:46.856 ******* 2021-02-08T07:10:40.207 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-08T07:10:40.208 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.252 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.054) 0:01:46.910 ******* 2021-02-08T07:10:40.270 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-08T07:10:40.272 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.307 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.054) 0:01:46.965 ******* 2021-02-08T07:10:40.318 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-08T07:10:40.320 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.375 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.068) 0:01:47.033 ******* 2021-02-08T07:10:40.383 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-08T07:10:40.384 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.423 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.047) 0:01:47.081 ******* 2021-02-08T07:10:40.432 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-08T07:10:40.432 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.473 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.050) 0:01:47.131 ******* 2021-02-08T07:10:40.475 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:40.481 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-08T07:10:40.481 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.539 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.065) 0:01:47.197 ******* 2021-02-08T07:10:40.546 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-08T07:10:40.547 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.596 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.056) 0:01:47.254 ******* 2021-02-08T07:10:40.610 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-08T07:10:40.612 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.650 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.054) 0:01:47.308 ******* 2021-02-08T07:10:40.659 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-08T07:10:40.661 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.698 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.047) 0:01:47.356 ******* 2021-02-08T07:10:40.726 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-08T07:10:40.728 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.767 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.064) 0:01:47.421 ******* 2021-02-08T07:10:40.790 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.826 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.062) 0:01:47.483 ******* 2021-02-08T07:10:40.832 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-08T07:10:40.833 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-02-08T07:10:40.872 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.045) 0:01:47.529 ******* 2021-02-08T07:10:40.880 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-08T07:10:40.880 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:40.926 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:40 +0000 (0:00:00.054) 0:01:47.584 ******* 2021-02-08T07:10:40.969 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:41.006 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:41 +0000 (0:00:00.079) 0:01:47.663 ******* 2021-02-08T07:10:41.044 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-08T07:10:41.046 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:41.085 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:41 +0000 (0:00:00.078) 0:01:47.742 ******* 2021-02-08T07:10:41.113 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-08T07:10:41.115 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:41.150 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:41 +0000 (0:00:00.065) 0:01:47.808 ******* 2021-02-08T07:10:41.258 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-08T07:10:41.259 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-02-08T07:10:41.296 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:41 +0000 (0:00:00.146) 0:01:47.954 ******* 2021-02-08T07:10:41.304 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-08T07:10:41.305 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-02-08T07:10:41.343 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:41 +0000 (0:00:00.046) 0:01:48.001 ******* 2021-02-08T07:10:41.344 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:41.360 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:41.361 INFO:teuthology.task.ansible.out:TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:41.398 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:41 +0000 (0:00:00.054) 0:01:48.056 ******* 2021-02-08T07:10:41.693 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-08T07:10:41.695 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-02-08T07:10:41.742 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:41 +0000 (0:00:00.344) 0:01:48.400 ******* 2021-02-08T07:10:41.760 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-08T07:10:41.761 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-02-08T07:10:41.802 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:41 +0000 (0:00:00.059) 0:01:48.460 ******* 2021-02-08T07:10:41.812 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-08T07:10:41.812 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:41.863 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:41 +0000 (0:00:00.060) 0:01:48.520 ******* 2021-02-08T07:10:42.148 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-08T07:10:42.149 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-02-08T07:10:42.190 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:42 +0000 (0:00:00.327) 0:01:48.848 ******* 2021-02-08T07:10:42.254 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-08T07:10:42.256 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-02-08T07:10:42.294 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:42 +0000 (0:00:00.103) 0:01:48.952 ******* 2021-02-08T07:10:42.541 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-08T07:10:42.542 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-02-08T07:10:42.581 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:42 +0000 (0:00:00.286) 0:01:49.239 ******* 2021-02-08T07:10:42.619 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-08T07:10:42.620 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-02-08T07:10:42.660 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:42 +0000 (0:00:00.079) 0:01:49.318 ******* 2021-02-08T07:10:42.720 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-08T07:10:42.721 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-08T07:10:42.760 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:42 +0000 (0:00:00.099) 0:01:49.417 ******* 2021-02-08T07:10:42.788 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-08T07:10:42.789 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:42.825 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:42 +0000 (0:00:00.065) 0:01:49.483 ******* 2021-02-08T07:10:42.854 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-08T07:10:42.856 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:42.892 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:42 +0000 (0:00:00.066) 0:01:49.550 ******* 2021-02-08T07:10:42.919 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-08T07:10:42.921 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:42.954 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:42 +0000 (0:00:00.062) 0:01:49.612 ******* 2021-02-08T07:10:42.981 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-08T07:10:42.983 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.019 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.064) 0:01:49.676 ******* 2021-02-08T07:10:43.028 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-08T07:10:43.030 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-02-08T07:10:43.072 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.052) 0:01:49.729 ******* 2021-02-08T07:10:43.100 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-08T07:10:43.101 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.136 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.064) 0:01:49.794 ******* 2021-02-08T07:10:43.174 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:43.176 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-08T07:10:43.208 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.072) 0:01:49.866 ******* 2021-02-08T07:10:43.226 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-08T07:10:43.228 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-08T07:10:43.279 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.070) 0:01:49.937 ******* 2021-02-08T07:10:43.287 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-08T07:10:43.288 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.289 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.291 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.291 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.313 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.315 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** ok: [smithi017.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi017.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.351 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.071) 0:01:50.008 ******* 2021-02-08T07:10:43.359 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.360 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.362 INFO:teuthology.task.ansible.out: skipping: [smithi017.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.363 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.409 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.058) 0:01:50.067 ******* 2021-02-08T07:10:43.416 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-08T07:10:43.417 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.455 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.045) 0:01:50.112 ******* 2021-02-08T07:10:43.484 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-08T07:10:43.485 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.523 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.068) 0:01:50.181 ******* 2021-02-08T07:10:43.551 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-08T07:10:43.552 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.589 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.065) 0:01:50.247 ******* 2021-02-08T07:10:43.617 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-08T07:10:43.618 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003189", "end": "2021-02-08 07:10:43.591480", "rc": 0, "start": "2021-02-08 07:10:43.588291", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_17c099650191e7aa47a7d9f502facca74f9b4d44/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_17c099650191e7aa47a7d9f502facca74f9b4d44/virtualenv/bin/teuthology-lock"]} TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.657 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.067) 0:01:50.315 ******* 2021-02-08T07:10:43.690 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.033) 0:01:50.348 ******* 2021-02-08T07:10:43.696 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-08T07:10:43.696 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.739 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.048) 0:01:50.397 ******* 2021-02-08T07:10:43.746 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:43.747 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-08T07:10:43.783 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.044) 0:01:50.441 ******* 2021-02-08T07:10:43.789 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-08T07:10:43.790 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.828 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.044) 0:01:50.486 ******* 2021-02-08T07:10:43.836 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-08T07:10:43.836 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.838 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.843 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.854 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.855 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-08T07:10:43.855 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.893 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.064) 0:01:50.551 ******* 2021-02-08T07:10:43.901 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-08T07:10:43.901 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.903 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi110.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-08T07:10:43.903 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.904 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.916 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.928 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.929 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.930 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:43.944 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-08T07:10:43.991 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:43 +0000 (0:00:00.097) 0:01:50.649 ******* 2021-02-08T07:10:44.196 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-08T07:10:44.197 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003511", "end": "2021-02-08 07:10:44.166569", "rc": 0, "start": "2021-02-08 07:10:44.163058", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_17c099650191e7aa47a7d9f502facca74f9b4d44/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_17c099650191e7aa47a7d9f502facca74f9b4d44/virtualenv/bin/teuthology-lock"]} 2021-02-08T07:10:44.246 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:44 +0000 (0:00:00.255) 0:01:50.904 ******* 2021-02-08T07:10:44.284 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-08T07:10:44.285 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:44.327 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:44 +0000 (0:00:00.080) 0:01:50.985 ******* 2021-02-08T07:10:44.354 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-08T07:10:44.355 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:44.390 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:44 +0000 (0:00:00.062) 0:01:51.047 ******* 2021-02-08T07:10:44.416 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-08T07:10:44.418 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:44.453 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:44 +0000 (0:00:00.063) 0:01:51.111 ******* 2021-02-08T07:10:44.824 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-08T07:10:44.826 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:10:44.862 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:44 +0000 (0:00:00.408) 0:01:51.519 ******* 2021-02-08T07:10:44.889 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False 2021-02-08T07:10:44.890 INFO:teuthology.task.ansible.out:"} 2021-02-08T07:10:44.926 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:44 +0000 (0:00:00.064) 0:01:51.584 ******* 2021-02-08T07:10:44.964 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-08T07:10:44.965 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item={'block_used': 13754792, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340316, 'size_available': 927089934336, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149544, 'block_size': 4096, 'inode_available': 60905432}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340316, "block_size": 4096, "block_total": 240095108, "block_used": 13754792, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905432, "inode_total": 61054976, "inode_used": 149544, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089934336, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} 2021-02-08T07:10:45.003 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.077) 0:01:51.661 ******* 2021-02-08T07:10:45.135 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-08T07:10:45.137 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009775", "end": "2021-02-08 07:10:45.134785", "rc": 0, "start": "2021-02-08 07:10:45.125010", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:45.177 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.173) 0:01:51.835 ******* 2021-02-08T07:10:45.226 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-08T07:10:45.228 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'block_used': 13754792, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340316, 'size_available': 927089934336, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149544, 'block_size': 4096, 'inode_available': 60905432}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340316, "block_size": 4096, "block_total": 240095108, "block_used": 13754792, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905432, "inode_total": 61054976, "inode_used": 149544, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089934336, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:45.271 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.093) 0:01:51.928 ******* 2021-02-08T07:10:45.298 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-08T07:10:45.334 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.063) 0:01:51.992 ******* 2021-02-08T07:10:45.382 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-08T07:10:45.383 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:10:45.384 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:45.424 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.090) 0:01:52.082 ******* 2021-02-08T07:10:45.452 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-08T07:10:45.454 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:45.492 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.067) 0:01:52.150 ******* 2021-02-08T07:10:45.530 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-08T07:10:45.531 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={'block_used': 13754781, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340327, 'size_available': 927089979392, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340327, "block_size": 4096, "block_total": 240095108, "block_used": 13754781, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089979392, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} 2021-02-08T07:10:45.572 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.079) 0:01:52.229 ******* 2021-02-08T07:10:45.713 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:45.715 INFO:teuthology.task.ansible.out:TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009766", "end": "2021-02-08 07:10:45.990183", "rc": 0, "start": "2021-02-08 07:10:45.980417", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:45.751 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.179) 0:01:52.409 ******* 2021-02-08T07:10:45.789 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-08T07:10:45.791 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'block_used': 13754781, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340327, 'size_available': 927089979392, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340327, "block_size": 4096, "block_total": 240095108, "block_used": 13754781, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089979392, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:45.831 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.079) 0:01:52.489 ******* 2021-02-08T07:10:45.858 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-08T07:10:45.897 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:45 +0000 (0:00:00.066) 0:01:52.555 ******* 2021-02-08T07:10:45.935 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-08T07:10:45.937 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:47.536 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3VRo0W4E5HlYbTaR5Ev8wS839IIaon6zG'], 'uuids': ['b743398c-a91a-4bb8-8ff7-44aa3814ff23']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.025599", "end": "2021-02-08 07:10:47.524606", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3VRo0W4E5HlYbTaR5Ev8wS839IIaon6zG"], "labels": [], "masters": [], "uuids": ["b743398c-a91a-4bb8-8ff7-44aa3814ff23"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:45.499007", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:47.548 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:47.549 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:48.176 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKR995xhhcxH6YABijHg23AhIIUBwvD2DP'], 'uuids': ['0686f864-629b-4abc-aaae-a1e2f1f92d00']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.092678", "end": "2021-02-08 07:10:48.444124", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKR995xhhcxH6YABijHg23AhIIUBwvD2DP"], "labels": [], "masters": [], "uuids": ["0686f864-629b-4abc-aaae-a1e2f1f92d00"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:46.351446", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:48.188 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:48.199 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:48.813 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3PwgklTiZ7IFLpnszbjio6sIFaI91PLwR'], 'uuids': ['1fba6af9-a801-4131-8917-33b93f27e617']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.129912", "end": "2021-02-08 07:10:48.799228", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3PwgklTiZ7IFLpnszbjio6sIFaI91PLwR"], "labels": [], "masters": [], "uuids": ["1fba6af9-a801-4131-8917-33b93f27e617"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:47.669316", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:48.814 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2SL', 'wwn-0x5000c500871b6e94'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2SL-part1', 'wwn-0x5000c500871b6e94-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b6e94', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J2SL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2SL", "wwn-0x5000c500871b6e94"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2SL-part1", "wwn-0x5000c500871b6e94-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2SL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b6e94"}}, "skip_reason": 2021-02-08T07:10:48.816 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-08T07:10:49.457 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKccoehz0JeM3L0LEHCvcvimLbeq2wWKKh'], 'uuids': ['22d99438-1a41-4022-b51d-18581bf24bfb']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.144399", "end": "2021-02-08 07:10:49.729281", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKccoehz0JeM3L0LEHCvcvimLbeq2wWKKh"], "labels": [], "masters": [], "uuids": ["22d99438-1a41-4022-b51d-18581bf24bfb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:48.584882", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:49.469 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QFRD', 'wwn-0x5000c50091dfc3c1'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1', 'wwn-0x5000c50091dfc3c1-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50091dfc3c1', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5QFRD', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD", "wwn-0x5000c50091dfc3c1"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1", "wwn-0x5000c50091dfc3c1-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QFRD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091dfc3c1"}}, "skip_reason": 2021-02-08T07:10:49.470 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-08T07:10:50.118 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-rnFCq6-3wSz-RC9S-1uxd-5jyN-qHUR-1vGaD6', 'nvme-INTEL_SSDPEDMD400G4_CVFT533000D8400BGN', 'nvme-nvme.8086-43564654353333303030443834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.187486", "end": "2021-02-08 07:10:50.116659", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-rnFCq6-3wSz-RC9S-1uxd-5jyN-qHUR-1vGaD6", "nvme-INTEL_SSDPEDMD400G4_CVFT533000D8400BGN", "nvme-nvme.8086-43564654353333303030443834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:48.929173", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-02-08T07:10:50.119 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:50.130 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:50.141 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:50.729 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-oy9rZp-nisM-aQkk-u2CU-fo2U-r9ne-M5pU2w', 'nvme-INTEL_SSDPED1D480GA_PHMB7476002T480DGN', 'nvme-nvme.8086-50484d42373437363030325434383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.156090", "end": "2021-02-08 07:10:51.004255", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-oy9rZp-nisM-aQkk-u2CU-fo2U-r9ne-M5pU2w", "nvme-INTEL_SSDPED1D480GA_PHMB7476002T480DGN", "nvme-nvme.8086-50484d42373437363030325434383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:49.848165", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-02-08T07:10:50.731 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:50.741 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:50.753 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:51.399 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf310lJTwG2VYPSMFLm24AeyWU3ifZdk1ED'], 'uuids': ['4558a27c-1c65-4f7a-9fc1-813a77db7ee0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.140833", "end": "2021-02-08 07:10:51.394312", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf310lJTwG2VYPSMFLm24AeyWU3ifZdk1ED"], "labels": [], "masters": [], "uuids": ["4558a27c-1c65-4f7a-9fc1-813a77db7ee0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:50.253479", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:51.411 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:51.422 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:52.117 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvK5LrxiTyNc3DfgKyndHxg9zsKOWFeEyDU'], 'uuids': ['a01399a4-693f-467c-8cd3-2fe0c8f0b1fc']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.252067", "end": "2021-02-08 07:10:52.395197", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvK5LrxiTyNc3DfgKyndHxg9zsKOWFeEyDU"], "labels": [], "masters": [], "uuids": ["a01399a4-693f-467c-8cd3-2fe0c8f0b1fc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:51.143130", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:52.128 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:52.140 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:52.693 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3XEX6lBZAji5Le8tMs7FBk626eSlOK0bF'], 'uuids': ['983c9b08-831f-40fe-ad82-9a7755dbae5b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.149877", "end": "2021-02-08 07:10:52.686004", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3XEX6lBZAji5Le8tMs7FBk626eSlOK0bF"], "labels": [], "masters": [], "uuids": ["983c9b08-831f-40fe-ad82-9a7755dbae5b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:51.536127", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:52.694 INFO:teuthology.task.ansible.out: skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:53.459 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKkzP48JztKRp0kANsn3zzcUXbMyRfeuIm'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.207942", "end": "2021-02-08 07:10:53.728216", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKkzP48JztKRp0kANsn3zzcUXbMyRfeuIm"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:52.520274", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:53.460 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:53.939 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3aPvSKgMJoDAX5NLMnVxIIxEefHXnaNXB'], 'uuids': ['5eebf673-2013-431c-afae-45933276c590']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.118676", "end": "2021-02-08 07:10:53.935954", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3aPvSKgMJoDAX5NLMnVxIIxEefHXnaNXB"], "labels": [], "masters": [], "uuids": ["5eebf673-2013-431c-afae-45933276c590"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:52.817278", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-08T07:10:53.986 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:53 +0000 (0:00:08.089) 0:02:00.644 ******* 2021-02-08T07:10:54.043 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-08T07:10:54.044 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:54.327 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3VRo0W4E5HlYbTaR5Ev8wS839IIaon6zG'], 'uuids': ['b743398c-a91a-4bb8-8ff7-44aa3814ff23']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.153220", "end": "2021-02-08 07:10:54.318679", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3VRo0W4E5HlYbTaR5Ev8wS839IIaon6zG"], "labels": [], "masters": [], "uuids": ["b743398c-a91a-4bb8-8ff7-44aa3814ff23"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:54.165459", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148384 s, 777 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148384 s, 777 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:54.339 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:54.340 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:54.602 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3PwgklTiZ7IFLpnszbjio6sIFaI91PLwR'], 'uuids': ['1fba6af9-a801-4131-8917-33b93f27e617']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.151775", "end": "2021-02-08 07:10:54.599309", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3PwgklTiZ7IFLpnszbjio6sIFaI91PLwR"], "labels": [], "masters": [], "uuids": ["1fba6af9-a801-4131-8917-33b93f27e617"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:54.447534", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147314 s, 783 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147314 s, 783 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:54.614 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2SL', 'wwn-0x5000c500871b6e94'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2SL-part1', 'wwn-0x5000c500871b6e94-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b6e94', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J2SL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2SL", "wwn-0x5000c500871b6e94"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2SL-part1", "wwn-0x5000c500871b6e94-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2SL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b6e94"}}, "skip_reason": 2021-02-08T07:10:54.616 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-08T07:10:54.751 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKMCucdaYy64OJ15RswiJhBV7hamfqaLnS'], 'uuids': ['c94b8c98-4416-4c09-9cda-e6f0ac579cf6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.171185", "end": "2021-02-08 07:10:55.025648", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKMCucdaYy64OJ15RswiJhBV7hamfqaLnS"], "labels": [], "masters": [], "uuids": ["c94b8c98-4416-4c09-9cda-e6f0ac579cf6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:53.854463", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk u 2021-02-08T07:10:54.753 INFO:teuthology.task.ansible.out:sing fdisk or", "other utilities."]} 2021-02-08T07:10:54.802 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:54 +0000 (0:00:00.815) 0:02:01.460 ******* 2021-02-08T07:10:54.809 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-08T07:10:54.809 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-rnFCq6-3wSz-RC9S-1uxd-5jyN-qHUR-1vGaD6', 'nvme-INTEL_SSDPEDMD400G4_CVFT533000D8400BGN', 'nvme-nvme.8086-43564654353333303030443834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.055535", "end": "2021-02-08 07:10:54.783775", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-rnFCq6-3wSz-RC9S-1uxd-5jyN-qHUR-1vGaD6", "nvme-INTEL_SSDPEDMD400G4_CVFT533000D8400BGN", "nvme-nvme.8086-43564654353333303030443834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:54.728240", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0506422 s, 2.3 GB/s", "std 2021-02-08T07:10:54.811 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0506422 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:54.811 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:54.812 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:54.846 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:55.061 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf310lJTwG2VYPSMFLm24AeyWU3ifZdk1ED'], 'uuids': ['4558a27c-1c65-4f7a-9fc1-813a77db7ee0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.140615", "end": "2021-02-08 07:10:55.059622", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf310lJTwG2VYPSMFLm24AeyWU3ifZdk1ED"], "labels": [], "masters": [], "uuids": ["4558a27c-1c65-4f7a-9fc1-813a77db7ee0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:54.919007", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137625 s, 838 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137625 s, 838 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:55.083 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:55.084 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKR995xhhcxH6YABijHg23AhIIUBwvD2DP'], 'uuids': ['0686f864-629b-4abc-aaae-a1e2f1f92d00']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.100055", "end": "2021-02-08 07:10:55.348101", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKR995xhhcxH6YABijHg23AhIIUBwvD2DP"], "labels": [], "masters": [], "uuids": ["0686f864-629b-4abc-aaae-a1e2f1f92d00"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:55.248046", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0973365 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0973365 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:55.086 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:55.110 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:55.123 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:55.343 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKccoehz0JeM3L0LEHCvcvimLbeq2wWKKh'], 'uuids': ['22d99438-1a41-4022-b51d-18581bf24bfb']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.106310", "end": "2021-02-08 07:10:55.618438", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKccoehz0JeM3L0LEHCvcvimLbeq2wWKKh"], "labels": [], "masters": [], "uuids": ["22d99438-1a41-4022-b51d-18581bf24bfb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:55.512128", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.10183 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.10183 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:55.355 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QFRD', 'wwn-0x5000c50091dfc3c1'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1', 'wwn-0x5000c50091dfc3c1-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50091dfc3c1', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5QFRD', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD", "wwn-0x5000c50091dfc3c1"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1", "wwn-0x5000c50091dfc3c1-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QFRD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091dfc3c1"}}, "skip_reason": 2021-02-08T07:10:55.357 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-08T07:10:55.367 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3XEX6lBZAji5Le8tMs7FBk626eSlOK0bF'], 'uuids': ['983c9b08-831f-40fe-ad82-9a7755dbae5b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.151492", "end": "2021-02-08 07:10:55.356880", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3XEX6lBZAji5Le8tMs7FBk626eSlOK0bF"], "labels": [], "masters": [], "uuids": ["983c9b08-831f-40fe-ad82-9a7755dbae5b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:55.205388", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147876 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147876 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:55.379 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:55.526 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-oy9rZp-nisM-aQkk-u2CU-fo2U-r9ne-M5pU2w', 'nvme-INTEL_SSDPED1D480GA_PHMB7476002T480DGN', 'nvme-nvme.8086-50484d42373437363030325434383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.049177", "end": "2021-02-08 07:10:55.800031", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-oy9rZp-nisM-aQkk-u2CU-fo2U-r9ne-M5pU2w", "nvme-INTEL_SSDPED1D480GA_PHMB7476002T480DGN", "nvme-nvme.8086-50484d42373437363030325434383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:55.750854", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0458599 s, 2.5 GB/s", "stderr_lines": [" 2021-02-08T07:10:55.527 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0458599 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:55.538 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:55.549 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:55.655 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3aPvSKgMJoDAX5NLMnVxIIxEefHXnaNXB'], 'uuids': ['5eebf673-2013-431c-afae-45933276c590']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.152935", "end": "2021-02-08 07:10:55.646161", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-DfKZmLi1tuAr5wLDzKdFaaS0AxGcTUf3aPvSKgMJoDAX5NLMnVxIIxEefHXnaNXB"], "labels": [], "masters": [], "uuids": ["5eebf673-2013-431c-afae-45933276c590"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:55.493226", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149846 s, 770 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149846 s, 770 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:55.704 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:55 +0000 (0:00:00.901) 0:02:02.361 ******* 2021-02-08T07:10:55.774 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvK5LrxiTyNc3DfgKyndHxg9zsKOWFeEyDU'], 'uuids': ['a01399a4-693f-467c-8cd3-2fe0c8f0b1fc']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.105391", "end": "2021-02-08 07:10:56.041916", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvK5LrxiTyNc3DfgKyndHxg9zsKOWFeEyDU"], "labels": [], "masters": [], "uuids": ["a01399a4-693f-467c-8cd3-2fe0c8f0b1fc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:55.936525", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.102516 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.102516 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:55.786 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:55.786 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:56.007 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKkzP48JztKRp0kANsn3zzcUXbMyRfeuIm'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.106533", "end": "2021-02-08 07:10:56.277093", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKkzP48JztKRp0kANsn3zzcUXbMyRfeuIm"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:56.170560", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.102106 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.102106 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:56.008 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-08T07:10:56.009 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.165761", "end": "2021-02-08 07:10:56.000654", "rc": 0, "start": "2021-02-08 07:10:55.834893", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:56.050 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.346) 0:02:02.707 ******* 2021-02-08T07:10:56.056 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-08T07:10:56.057 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:56.236 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKMCucdaYy64OJ15RswiJhBV7hamfqaLnS'], 'uuids': ['c94b8c98-4416-4c09-9cda-e6f0ac579cf6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.111416", "end": "2021-02-08 07:10:56.506663", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-b1JdOy3YQbCLMSDf2u8CM8fKcTkJLQvKMCucdaYy64OJ15RswiJhBV7hamfqaLnS"], "labels": [], "masters": [], "uuids": ["c94b8c98-4416-4c09-9cda-e6f0ac579cf6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-08 07:10:56.395247", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.108555 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.108555 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:56.247 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-08T07:10:56.248 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.057907", "end": "2021-02-08 07:10:56.242860", "rc": 0, "start": "2021-02-08 07:10:56.184953", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:56.284 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.234) 0:02:02.942 ******* 2021-02-08T07:10:56.323 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.039) 0:02:02.981 ******* 2021-02-08T07:10:56.331 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-08T07:10:56.371 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.047) 0:02:03.029 ******* 2021-02-08T07:10:56.408 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-08T07:10:56.410 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:56.449 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.078) 0:02:03.107 ******* 2021-02-08T07:10:56.476 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-08T07:10:56.477 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:56.511 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.062) 0:02:03.169 ******* 2021-02-08T07:10:56.538 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-08T07:10:56.540 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:56.577 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.065) 0:02:03.235 ******* 2021-02-08T07:10:56.594 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-08T07:10:56.595 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.137228", "end": "2021-02-08 07:10:56.856216", "rc": 0, "start": "2021-02-08 07:10:56.718988", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:10:56.629 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.052) 0:02:03.287 ******* 2021-02-08T07:10:56.635 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-08T07:10:56.635 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:56.672 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.043) 0:02:03.330 ******* 2021-02-08T07:10:56.796 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-08T07:10:56.797 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.052002", "end": "2021-02-08 07:10:57.072768", "rc": 0, "start": "2021-02-08 07:10:57.020766", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-08T07:10:56.832 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.159) 0:02:03.490 ******* 2021-02-08T07:10:56.876 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.044) 0:02:03.534 ******* 2021-02-08T07:10:56.883 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:56.885 INFO:teuthology.task.ansible.out:TASK [testnode : Set root disk] ************************************************ skipping: [smithi017.front.sepia.ceph.com] => (item={'block_used': 13754792, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340316, 'size_available': 927089934336, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149544, 'block_size': 4096, 'inode_available': 60905432}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340316, "block_size": 4096, "block_total": 240095108, "block_used": 13754792, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905432, "inode_total": 61054976, "inode_used": 149544, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089934336, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:56.923 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.046) 0:02:03.581 ******* 2021-02-08T07:10:56.930 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-08T07:10:56.972 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:56 +0000 (0:00:00.049) 0:02:03.630 ******* 2021-02-08T07:10:56.980 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.017 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.044) 0:02:03.675 ******* 2021-02-08T07:10:57.023 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-08T07:10:57.024 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.063 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.046) 0:02:03.721 ******* 2021-02-08T07:10:57.069 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:57.069 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.107 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.043) 0:02:03.764 ******* 2021-02-08T07:10:57.112 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-08T07:10:57.114 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.155 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.048) 0:02:03.813 ******* 2021-02-08T07:10:57.161 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:57.163 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.199 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.043) 0:02:03.857 ******* 2021-02-08T07:10:57.206 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-08T07:10:57.208 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.245 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.046) 0:02:03.903 ******* 2021-02-08T07:10:57.251 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-08T07:10:57.252 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.288 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.042) 0:02:03.946 ******* 2021-02-08T07:10:57.295 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-08T07:10:57.297 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.332 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.043) 0:02:03.990 ******* 2021-02-08T07:10:57.337 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-08T07:10:57.339 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.377 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.044) 0:02:04.035 ******* 2021-02-08T07:10:57.404 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-08T07:10:57.405 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.441 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.064) 0:02:04.099 ******* 2021-02-08T07:10:57.470 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-08T07:10:57.471 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-08T07:10:57.510 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.068) 0:02:04.168 ******* 2021-02-08T07:10:57.549 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-08T07:10:57.550 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'block_used': 13754781, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340327, 'size_available': 927089979392, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340327, "block_size": 4096, "block_total": 240095108, "block_used": 13754781, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089979392, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.587 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.077) 0:02:04.245 ******* 2021-02-08T07:10:57.615 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-08T07:10:57.617 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.653 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.066) 0:02:04.311 ******* 2021-02-08T07:10:57.682 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:57.682 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:57.683 INFO:teuthology.task.ansible.out: skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.717 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.063) 0:02:04.375 ******* 2021-02-08T07:10:57.744 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:10:57.746 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.780 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.062) 0:02:04.438 ******* 2021-02-08T07:10:57.807 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-08T07:10:57.809 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.845 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.065) 0:02:04.503 ******* 2021-02-08T07:10:57.888 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-08T07:10:57.889 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.924 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.079) 0:02:04.582 ******* 2021-02-08T07:10:57.952 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-08T07:10:57.954 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:10:57.988 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:57 +0000 (0:00:00.064) 0:02:04.646 ******* 2021-02-08T07:10:58.017 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:58.018 INFO:teuthology.task.ansible.out:TASK [testnode : Create volume group(s)] *************************************** changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-08T07:10:58.064 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:58 +0000 (0:00:00.075) 0:02:04.722 ******* 2021-02-08T07:10:58.065 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:58.317 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-08T07:10:58.356 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:58 +0000 (0:00:00.292) 0:02:05.014 ******* 2021-02-08T07:10:58.641 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:58.642 INFO:teuthology.task.ansible.out:TASK [testnode : Create logical volume(s)] ************************************* 2021-02-08T07:10:58.643 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:58.739 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:58.887 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:59.015 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:59.117 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:59.279 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:59.342 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:59.540 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:59.551 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:59.590 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:59 +0000 (0:00:01.234) 0:02:06.248 ******* 2021-02-08T07:10:59.734 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-08T07:10:59.735 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-08T07:10:59.773 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:59 +0000 (0:00:00.182) 0:02:06.431 ******* 2021-02-08T07:10:59.790 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-08T07:10:59.792 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-08T07:10:59.836 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:10:59 +0000 (0:00:00.062) 0:02:06.493 ******* 2021-02-08T07:10:59.837 INFO:teuthology.task.ansible.out: 2021-02-08T07:10:59.910 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-08T07:10:59.911 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-08T07:10:59.975 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-08T07:10:59.976 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-08T07:11:00.010 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.174) 0:02:06.668 ******* 2021-02-08T07:11:00.027 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-08T07:11:00.028 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-08T07:11:00.136 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-08T07:11:00.158 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-08T07:11:00.254 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-08T07:11:00.266 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-08T07:11:00.267 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:00.305 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.294) 0:02:06.963 ******* 2021-02-08T07:11:00.343 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-08T07:11:00.344 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:00.381 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.075) 0:02:07.039 ******* 2021-02-08T07:11:00.387 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-08T07:11:00.389 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-08T07:11:00.411 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-08T07:11:00.412 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:00.447 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.065) 0:02:07.105 ******* 2021-02-08T07:11:00.496 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-08T07:11:00.496 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-08T07:11:00.508 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:00.548 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.101) 0:02:07.206 ******* 2021-02-08T07:11:00.575 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-08T07:11:00.577 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:11:00.613 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.063) 0:02:07.270 ******* 2021-02-08T07:11:00.647 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.034) 0:02:07.305 ******* 2021-02-08T07:11:00.652 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-08T07:11:00.654 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:00.694 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.047) 0:02:07.352 ******* 2021-02-08T07:11:00.702 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-08T07:11:00.703 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-08T07:11:00.745 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.050) 0:02:07.402 ******* 2021-02-08T07:11:00.824 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-08T07:11:00.825 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:11:00.862 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.117) 0:02:07.520 ******* 2021-02-08T07:11:00.900 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-08T07:11:00.902 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-08T07:11:00.936 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:00 +0000 (0:00:00.073) 0:02:07.594 ******* 2021-02-08T07:11:01.221 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-08T07:11:01.223 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true} 2021-02-08T07:11:01.263 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:01 +0000 (0:00:00.326) 0:02:07.920 ******* 2021-02-08T07:11:01.720 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-08T07:11:01.721 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-08T07:11:01.758 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:01 +0000 (0:00:00.495) 0:02:08.415 ******* 2021-02-08T07:11:01.795 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-08T07:11:01.796 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:01.831 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:01 +0000 (0:00:00.073) 0:02:08.489 ******* 2021-02-08T07:11:01.857 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-08T07:11:01.859 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:01.893 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:01 +0000 (0:00:00.061) 0:02:08.551 ******* 2021-02-08T07:11:01.930 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-08T07:11:01.931 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:01.965 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:01 +0000 (0:00:00.071) 0:02:08.623 ******* 2021-02-08T07:11:01.991 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-08T07:11:01.992 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:02.026 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.061) 0:02:08.684 ******* 2021-02-08T07:11:02.152 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-08T07:11:02.153 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true} 2021-02-08T07:11:02.190 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.163) 0:02:08.848 ******* 2021-02-08T07:11:02.198 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-08T07:11:02.199 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-08T07:11:02.237 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.046) 0:02:08.894 ******* 2021-02-08T07:11:02.368 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-08T07:11:02.370 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:11:02.404 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.167) 0:02:09.062 ******* 2021-02-08T07:11:02.411 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-08T07:11:02.411 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-08T07:11:02.449 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.044) 0:02:09.107 ******* 2021-02-08T07:11:02.475 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-08T07:11:02.476 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:02.514 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.064) 0:02:09.172 ******* 2021-02-08T07:11:02.541 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-08T07:11:02.542 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:02.584 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.069) 0:02:09.242 ******* 2021-02-08T07:11:02.610 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:02.660 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.076) 0:02:09.318 ******* 2021-02-08T07:11:02.689 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-08T07:11:02.690 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:02.725 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.065) 0:02:09.383 ******* 2021-02-08T07:11:02.732 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-08T07:11:02.732 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-08T07:11:02.771 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.045) 0:02:09.428 ******* 2021-02-08T07:11:02.893 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-08T07:11:02.895 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-08T07:11:02.937 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:02 +0000 (0:00:00.166) 0:02:09.595 ******* 2021-02-08T07:11:03.082 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-08T07:11:03.084 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:11:03.121 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:03 +0000 (0:00:00.184) 0:02:09.779 ******* 2021-02-08T07:11:03.128 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-08T07:11:03.129 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-08T07:11:03.171 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:03 +0000 (0:00:00.049) 0:02:09.829 ******* 2021-02-08T07:11:03.469 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-08T07:11:03.471 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-08T07:11:03.508 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:03 +0000 (0:00:00.336) 0:02:10.166 ******* 2021-02-08T07:11:03.806 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-08T07:11:03.807 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-08T07:11:03.856 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:03 +0000 (0:00:00.347) 0:02:10.514 ******* 2021-02-08T07:11:04.106 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:11:04.145 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:04 +0000 (0:00:00.288) 0:02:10.803 ******* 2021-02-08T07:11:04.334 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.045062", "end": "2021-02-08 07:11:04.336791", "rc": 0, "start": "2021-02-08 07:11:04.291729", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-08T07:11:04.377 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:04 +0000 (0:00:00.231) 0:02:11.034 ******* 2021-02-08T07:11:04.425 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-08T07:11:04.427 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:04.461 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:04 +0000 (0:00:00.084) 0:02:11.119 ******* 2021-02-08T07:11:04.488 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-08T07:11:04.489 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:04.523 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:04 +0000 (0:00:00.062) 0:02:11.181 ******* 2021-02-08T07:11:04.764 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-08T07:11:04.765 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:11:04.805 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:04 +0000 (0:00:00.281) 0:02:11.463 ******* 2021-02-08T07:11:04.981 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-08T07:11:04.982 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047341", "end": "2021-02-08 07:11:05.250744", "rc": 0, "start": "2021-02-08 07:11:05.203403", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-08T07:11:05.042 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:05 +0000 (0:00:00.237) 0:02:11.700 ******* 2021-02-08T07:11:05.083 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-08T07:11:05.085 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:05.132 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:05 +0000 (0:00:00.090) 0:02:11.790 ******* 2021-02-08T07:11:05.170 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-08T07:11:05.171 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:05.205 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:05 +0000 (0:00:00.072) 0:02:11.863 ******* 2021-02-08T07:11:05.420 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-08T07:11:05.422 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:11:05.461 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:05 +0000 (0:00:00.255) 0:02:12.119 ******* 2021-02-08T07:11:05.487 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-08T07:11:05.489 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:05.523 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:05 +0000 (0:00:00.062) 0:02:12.181 ******* 2021-02-08T07:11:05.845 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-08T07:11:05.846 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-02-08T07:11:05.885 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:05 +0000 (0:00:00.362) 0:02:12.543 ******* 2021-02-08T07:11:05.912 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-08T07:11:05.913 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:06.069 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.183) 0:02:12.727 ******* 2021-02-08T07:11:06.107 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-08T07:11:06.108 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:11:06.148 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.078) 0:02:12.806 ******* 2021-02-08T07:11:06.175 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-08T07:11:06.176 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:06.211 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.063) 0:02:12.869 ******* 2021-02-08T07:11:06.280 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-08T07:11:06.281 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-02-08 07:08:27 UTC", "ActiveEnterTimestampMonotonic": "25248783", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service basic.target sysinit.target system.slice -.mount systemd-journald.socket network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-02-08 07:08:21 UTC", "AssertTimestampMonotonic": "18820061", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-02-08 07:08:21 UTC", "ConditionTimestampMonotonic": "18820061", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-02-08T07:11:06.283 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1749", "ExecMainStartTimestamp": "Mon 2021-02-08 07:08:27 UTC", "ExecMainStartTimestampMonotonic": "25248718", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-02-08 07:08:21 UTC", "InactiveExitTimestampMonotonic": "18820824", "InvocationID": "f2b17a8df51045aa886997fae577ff84", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-02-08T07:11:06.284 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1749", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-02-08 07:08:27 UTC", "S 2021-02-08T07:11:06.285 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "25248783", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-02-08 07:08:27 UTC", "WatchdogTimestampMonotonic": "25248779", "WatchdogUSec": "0"}} 2021-02-08T07:11:06.319 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.108) 0:02:12.977 ******* 2021-02-08T07:11:06.346 INFO:teuthology.task.ansible.out: 2021-02-08T07:11:06.348 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-08T07:11:06.385 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.065) 0:02:13.043 ******* 2021-02-08T07:11:06.413 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:11:06.414 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:06.449 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.063) 0:02:13.107 ******* 2021-02-08T07:11:06.476 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-08T07:11:06.477 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:06.513 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.063) 0:02:13.171 ******* 2021-02-08T07:11:06.529 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-08T07:11:06.530 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-02-08T07:11:06.569 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.056) 0:02:13.227 ******* 2021-02-08T07:11:06.597 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-08T07:11:06.599 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:06.634 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.064) 0:02:13.291 ******* 2021-02-08T07:11:06.859 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-08T07:11:06.860 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-02-08 07:08:42 UTC", "ActiveEnterTimestampMonotonic": "27633386", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service system.slice network.target systemd-journald.socket -.mount basic.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-02-08 07:08:35 UTC", "AssertTimestampMonotonic": "21020125", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-02-08 07:08:35 UTC", "ConditionTimestampMonotonic": "21020125", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-02-08T07:11:06.861 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1771", "ExecMainStartTimestamp": "Mon 2021-02-08 07:08:42 UTC", "ExecMainStartTimestampMonotonic": "27633367", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-02-08 07:08:35 UTC", "InactiveExitTimestampMonotonic": "21020986", "InvocationID": "e1e7951adab94a76a4f2311ebf3ced6a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128028", "LimitNPROCSoft": "128028", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-02-08T07:11:06.863 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "128028", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1771", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-02-08 07:08:42 UTC", "S 2021-02-08T07:11:06.865 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "27633386", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-02-08 07:08:42 UTC", "WatchdogTimestampMonotonic": "27633384", "WatchdogUSec": "0"}} 2021-02-08T07:11:06.897 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.263) 0:02:13.555 ******* 2021-02-08T07:11:06.925 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:11:06.927 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-08T07:11:06.968 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:06 +0000 (0:00:00.071) 0:02:13.626 ******* 2021-02-08T07:11:06.996 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:11:06.997 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:07.036 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:07 +0000 (0:00:00.067) 0:02:13.694 ******* 2021-02-08T07:11:07.063 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-08T07:11:07.064 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:07.101 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:07 +0000 (0:00:00.064) 0:02:13.758 ******* 2021-02-08T07:11:07.408 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-08T07:11:07.409 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:11:07.446 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:07 +0000 (0:00:00.345) 0:02:14.104 ******* 2021-02-08T07:11:07.473 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-08T07:11:07.474 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:07.513 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:07 +0000 (0:00:00.066) 0:02:14.170 ******* 2021-02-08T07:11:07.643 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-08T07:11:07.645 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-02-08T07:11:07.680 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:07 +0000 (0:00:00.167) 0:02:14.338 ******* 2021-02-08T07:11:07.988 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-08T07:11:07.990 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-02-08T07:11:08.026 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:08 +0000 (0:00:00.346) 0:02:14.684 ******* 2021-02-08T07:11:08.033 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-08T07:11:08.033 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:11:08.073 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:08 +0000 (0:00:00.046) 0:02:14.731 ******* 2021-02-08T07:11:08.101 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-08T07:11:08.102 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:08.146 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:08 +0000 (0:00:00.073) 0:02:14.804 ******* 2021-02-08T07:11:08.299 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-08T07:11:08.301 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-02-08T07:11:08.335 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:08 +0000 (0:00:00.188) 0:02:14.992 ******* 2021-02-08T07:11:08.804 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-08T07:11:08.806 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-02-08T07:11:08.839 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:08 +0000 (0:00:00.504) 0:02:15.497 ******* 2021-02-08T07:11:08.963 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-08T07:11:08.965 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-08T07:11:09.001 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:08 +0000 (0:00:00.162) 0:02:15.659 ******* 2021-02-08T07:11:09.515 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-08T07:11:09.517 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false} 2021-02-08T07:11:09.553 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:09 +0000 (0:00:00.552) 0:02:16.211 ******* 2021-02-08T07:11:09.580 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-08T07:11:09.582 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-02-08T07:11:09.617 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:09 +0000 (0:00:00.064) 0:02:16.275 ******* 2021-02-08T07:11:09.623 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-08T07:11:09.624 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-08T07:11:09.662 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:09 +0000 (0:00:00.044) 0:02:16.319 ******* 2021-02-08T07:11:09.761 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-08T07:11:09.762 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1612767846.6160278, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1612767846.6120298, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1612767846.6120298, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-02-08T07:11:09.799 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:09 +0000 (0:00:00.137) 0:02:16.457 ******* 2021-02-08T07:11:10.066 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-08T07:11:10.067 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-02-08T07:11:10.106 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.307) 0:02:16.764 ******* 2021-02-08T07:11:10.133 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-08T07:11:10.134 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-08T07:11:10.176 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.069) 0:02:16.833 ******* 2021-02-08T07:11:10.182 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-08T07:11:10.184 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false} 2021-02-08T07:11:10.219 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.042) 0:02:16.876 ******* 2021-02-08T07:11:10.224 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-08T07:11:10.226 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.264 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.045) 0:02:16.922 ******* 2021-02-08T07:11:10.271 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-08T07:11:10.272 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-02-08T07:11:10.308 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.043) 0:02:16.966 ******* 2021-02-08T07:11:10.387 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-08T07:11:10.388 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-08T07:11:10.424 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.116) 0:02:17.082 ******* 2021-02-08T07:11:10.451 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-08T07:11:10.452 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1612767898.7636724, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1612767898.7596722, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1612767898.7596722, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-02-08T07:11:10.489 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.064) 0:02:17.147 ******* 2021-02-08T07:11:10.557 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-08T07:11:10.559 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-08T07:11:10.594 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.105) 0:02:17.252 ******* 2021-02-08T07:11:10.621 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-08T07:11:10.623 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.658 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.063) 0:02:17.316 ******* 2021-02-08T07:11:10.665 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-08T07:11:10.665 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-02-08T07:11:10.703 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.044) 0:02:17.361 ******* 2021-02-08T07:11:10.710 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-08T07:11:10.710 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.711 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.712 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.723 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.759 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.055) 0:02:17.417 ******* 2021-02-08T07:11:10.765 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-08T07:11:10.767 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-08T07:11:10.804 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.044) 0:02:17.462 ******* 2021-02-08T07:11:10.811 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-08T07:11:10.811 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.813 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.824 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.824 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.826 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-08T07:11:10.827 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.861 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.057) 0:02:17.519 ******* 2021-02-08T07:11:10.894 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.032) 0:02:17.552 ******* 2021-02-08T07:11:10.901 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:11:10.902 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:10.940 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:10 +0000 (0:00:00.046) 0:02:17.598 ******* 2021-02-08T07:11:10.968 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-08T07:11:10.969 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.010 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.069) 0:02:17.668 ******* 2021-02-08T07:11:11.017 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-08T07:11:11.017 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-08T07:11:11.063 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.053) 0:02:17.721 ******* 2021-02-08T07:11:11.072 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-08T07:11:11.072 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.113 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.050) 0:02:17.771 ******* 2021-02-08T07:11:11.142 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-08T07:11:11.143 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.181 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.067) 0:02:17.839 ******* 2021-02-08T07:11:11.198 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-08T07:11:11.199 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-08T07:11:11.249 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.067) 0:02:17.906 ******* 2021-02-08T07:11:11.257 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-08T07:11:11.258 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.308 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.059) 0:02:17.966 ******* 2021-02-08T07:11:11.316 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-08T07:11:11.318 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.358 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.049) 0:02:18.016 ******* 2021-02-08T07:11:11.364 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-08T07:11:11.364 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.405 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.046) 0:02:18.063 ******* 2021-02-08T07:11:11.411 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-08T07:11:11.413 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.414 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.455 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.049) 0:02:18.112 ******* 2021-02-08T07:11:11.462 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-08T07:11:11.463 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.505 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.050) 0:02:18.163 ******* 2021-02-08T07:11:11.513 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-08T07:11:11.515 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.516 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.517 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.519 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.559 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.053) 0:02:18.217 ******* 2021-02-08T07:11:11.566 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-08T07:11:11.568 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.603 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.043) 0:02:18.261 ******* 2021-02-08T07:11:11.611 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-08T07:11:11.612 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.658 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.055) 0:02:18.316 ******* 2021-02-08T07:11:11.686 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-08T07:11:11.688 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.723 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.065) 0:02:18.381 ******* 2021-02-08T07:11:11.750 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-08T07:11:11.751 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.789 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.066) 0:02:18.447 ******* 2021-02-08T07:11:11.796 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-08T07:11:11.796 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011142", "end": "2021-02-08 07:11:11.755202", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-08 07:11:11.744060", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:11:11.819 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-08T07:11:11.820 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.857 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.068) 0:02:18.515 ******* 2021-02-08T07:11:11.885 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-08T07:11:11.886 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006082", "end": "2021-02-08 07:11:11.881894", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-08 07:11:11.875812", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.924 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.066) 0:02:18.582 ******* 2021-02-08T07:11:11.951 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-08T07:11:11.952 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:11.989 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:11 +0000 (0:00:00.065) 0:02:18.647 ******* 2021-02-08T07:11:12.006 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-08T07:11:12.008 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005501", "end": "2021-02-08 07:11:11.999881", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-08 07:11:11.994380", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-08T07:11:12.018 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-08T07:11:12.018 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:12.059 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:12 +0000 (0:00:00.069) 0:02:18.717 ******* 2021-02-08T07:11:12.086 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-08T07:11:12.087 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:12.122 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:12 +0000 (0:00:00.062) 0:02:18.780 ******* 2021-02-08T07:11:12.128 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-08T07:11:12.129 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.006736", "end": "2021-02-08 07:11:12.115324", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-08 07:11:12.108588", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:11:12.173 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:12 +0000 (0:00:00.051) 0:02:18.831 ******* 2021-02-08T07:11:12.201 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-08T07:11:12.239 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:12 +0000 (0:00:00.065) 0:02:18.896 ******* 2021-02-08T07:11:12.277 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-08T07:11:12.278 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.009277", "end": "2021-02-08 07:11:12.536110", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-08 07:11:12.526833", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:11:12.372 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-08T07:11:12.373 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-02-08T07:11:12.408 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-08T07:11:12.409 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005985", "end": "2021-02-08 07:11:12.656158", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-08 07:11:12.650173", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:11:12.504 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003547", "end": "2021-02-08 07:11:12.765975", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-08 07:11:12.762428", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-08T07:11:12.650 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.024264", "end": "2021-02-08 07:11:12.912013", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-08 07:11:12.887749", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:11:12.686 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:12 +0000 (0:00:00.447) 0:02:19.344 ******* 2021-02-08T07:11:12.712 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-08T07:11:12.748 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:12 +0000 (0:00:00.062) 0:02:19.406 ******* 2021-02-08T07:11:12.878 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-08T07:11:12.880 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-02-08T07:11:12.986 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-02-08T07:11:13.020 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:13 +0000 (0:00:00.271) 0:02:19.678 ******* 2021-02-08T07:11:13.056 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:13 +0000 (0:00:00.036) 0:02:19.714 ******* 2021-02-08T07:11:13.757 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi110.front.sepia.ceph.com] 2021-02-08T07:11:13.810 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] 2021-02-08T07:11:13.886 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:13 +0000 (0:00:00.829) 0:02:20.544 ******* 2021-02-08T07:11:13.935 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:13 +0000 (0:00:00.048) 0:02:20.592 ******* 2021-02-08T07:11:13.941 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-08T07:11:13.943 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-02-08T07:11:13.982 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:13 +0000 (0:00:00.047) 0:02:20.640 ******* 2021-02-08T07:11:13.989 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-02-08T07:11:14.028 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:14 +0000 (0:00:00.045) 0:02:20.686 ******* 2021-02-08T07:11:14.034 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-08T07:11:14.036 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:14.073 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:14 +0000 (0:00:00.044) 0:02:20.731 ******* 2021-02-08T07:11:14.079 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:14.121 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:14 +0000 (0:00:00.047) 0:02:20.778 ******* 2021-02-08T07:11:14.127 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-08T07:11:14.128 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-02-08T07:11:14.168 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:14 +0000 (0:00:00.047) 0:02:20.826 ******* 2021-02-08T07:11:14.175 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-02-08T07:11:14.216 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:14 +0000 (0:00:00.047) 0:02:20.874 ******* 2021-02-08T07:11:15.068 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-08T07:11:15.069 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1612767960, "cache_updated": false, "changed": false} 2021-02-08T07:11:15.105 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:15 +0000 (0:00:00.889) 0:02:21.763 ******* 2021-02-08T07:11:15.132 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-08T07:11:15.134 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:15.172 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:15 +0000 (0:00:00.067) 0:02:21.830 ******* 2021-02-08T07:11:15.179 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-08T07:11:15.179 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1612767914, "cache_updated": false, "changed": false} 2021-02-08T07:11:15.219 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:15 +0000 (0:00:00.046) 0:02:21.877 ******* 2021-02-08T07:11:15.227 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-08T07:11:15.228 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-08T07:11:15.266 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:15 +0000 (0:00:00.046) 0:02:21.924 ******* 2021-02-08T07:11:15.272 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-08T07:11:15.273 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-08T07:11:15.310 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:15 +0000 (0:00:00.044) 0:02:21.968 ******* 2021-02-08T07:11:15.339 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-08T07:11:15.340 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-08T07:11:15.379 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:15 +0000 (0:00:00.068) 0:02:22.037 ******* 2021-02-08T07:11:15.417 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-08T07:11:15.417 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:11:15.533 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:11:15.535 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:11:15.576 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:15 +0000 (0:00:00.196) 0:02:22.234 ******* 2021-02-08T07:11:15.645 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-08T07:11:15.689 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:15 +0000 (0:00:00.112) 0:02:22.347 ******* 2021-02-08T07:11:15.905 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-08T07:11:15.907 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612768275.6188064-256433223260352/source", "state": "file", "uid": 0} 2021-02-08T07:11:16.033 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612768275.7298098-116845463970945/source", "state": "file", "uid": 0} 2021-02-08T07:11:16.222 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612768275.9079292-123176054248238/source", "state": "file", "uid": 0} 2021-02-08T07:11:16.263 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:16 +0000 (0:00:00.573) 0:02:22.920 ******* 2021-02-08T07:11:16.346 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612768276.0416384-10547016416089/source", "state": "file", "uid": 0} 2021-02-08T07:11:16.391 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:16 +0000 (0:00:00.128) 0:02:23.049 ******* 2021-02-08T07:11:19.822 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-08T07:11:19.824 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-ofHT1n-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-ofHT1n-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-02-08T07:11:19.825 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-08T07:11:19.861 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:19 +0000 (0:00:03.469) 0:02:26.519 ******* 2021-02-08T07:11:19.962 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-mzKpWD-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-mzKpWD-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-02-08T07:11:19.963 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-08T07:11:20.001 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:19 +0000 (0:00:00.140) 0:02:26.659 ******* 2021-02-08T07:11:20.008 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-08T07:11:20.010 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-08T07:11:20.049 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:20 +0000 (0:00:00.047) 0:02:26.707 ******* 2021-02-08T07:11:20.139 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-08T07:11:20.189 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:20 +0000 (0:00:00.139) 0:02:26.847 ******* 2021-02-08T07:11:20.400 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-02-08T07:11:20.401 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612768280.088169-262830422693459/source", "state": "file", "uid": 0} 2021-02-08T07:11:20.443 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:20 +0000 (0:00:00.253) 0:02:27.101 ******* 2021-02-08T07:11:20.513 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612768280.2233543-256521605660006/source", "state": "file", "uid": 0} 2021-02-08T07:11:20.556 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:20 +0000 (0:00:00.112) 0:02:27.213 ******* 2021-02-08T07:11:20.709 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.119551", "end": "2021-02-08 07:11:20.974117", "rc": 0, "start": "2021-02-08 07:11:20.854566", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:11:20.747 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:20 +0000 (0:00:00.191) 0:02:27.405 ******* 2021-02-08T07:11:20.818 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.123511", "end": "2021-02-08 07:11:20.823525", "rc": 0, "start": "2021-02-08 07:11:20.700014", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-08T07:11:20.860 INFO:teuthology.task.ansible.out:Monday 08 February 2021 07:11:20 +0000 (0:00:00.112) 0:02:27.518 ******* 2021-02-08T07:11:23.947 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-08T07:11:23.948 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target systemd-journald.socket network-online.target firewalld.service containerd.service docker.socket sysinit.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-02-08T07:11:23.949 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "12802 2021-02-08T07:11:23.950 INFO:teuthology.task.ansible.out:8", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket sysinit.target system.slice", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-02-08 07:10:21 UTC", "StateChangeTimestampMonotonic": "126499581", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-02-08T07:11:23.952 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-08T07:11:24.265 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "containerd.service docker.socket systemd-journald.socket firewalld.service system.slice sysinit.target network-online.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-02-08T07:11:24.266 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "12805 2021-02-08T07:11:24.268 INFO:teuthology.task.ansible.out:3", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target docker.socket system.slice", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-02-08 07:10:19 UTC", "StateChangeTimestampMonotonic": "136512164", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-02-08T07:11:24.270 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-08T07:11:24.379 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-02-08T07:11:24.384 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-02-08T07:11:24.388 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-02-08T07:11:24.389 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi017.front.sepia.ceph.com : ok=126 changed=36 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi110.front.sepia.ceph.com : ok=126 changed=36 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Monday 08 February 2021 07:11:24 +0000 (0:00:03.528) 0:02:31.046 ******* =============================================================================== users : Create all admin users with sudo access. ----------------------- 18.35s users : Update authorized_keys using the keys repo --------------------- 16.76s testnode : Install packages --------------------------------------------- 8.19s testnode : Zap all non-root disks --------------------------------------- 8.09s testnode : Ensure packages are not present. ----------------------------- 7.44s users : Remove revoked users -------------------------------------------- 4.09s container-host : Restart docker service --------------------------------- 3.53s container-host : Install registries-conf-ctl ---------------------------- 3.47s Gathering Facts --------------------------------------------------------- 1.72s users : Update authorized_keys for each user with literal keys ---------- 1.65s testnode : Update apt cache. -------------------------------------------- 1.41s common : Update apt cache ----------------------------------------------- 1.30s testnode : Create logical volume(s) ------------------------------------- 1.23s testnode : Install python-apt ------------------------------------------- 1.06s testnode : Blow away lingering OSD data and FSIDs ----------------------- 0.90s container-host : Install container packages ----------------------------- 0.89s Gathering Facts --------------------------------------------------------- 0.83s Gathering Facts --------------------------------------------------------- 0.81s common : Make sure smartmontools is installed 2021-02-08T07:11:24.390 INFO:teuthology.task.ansible.out:--------------------------- 0.77s testnode : Install non aarch64 packages --------------------------------- 0.66s 2021-02-08T07:11:24.721 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com')] 2021-02-08T07:11:24.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-08T07:11:24.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-08T07:11:24.801 DEBUG:teuthology.orchestra.run.smithi017:> true 2021-02-08T07:11:25.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-08T07:11:25.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-08T07:11:25.080 DEBUG:teuthology.orchestra.run.smithi110:> true 2021-02-08T07:11:25.271 INFO:teuthology.run_tasks:Running task clock... 2021-02-08T07:11:25.279 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-08T07:11:25.279 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-08T07:11:25.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-08T07:11:25.342 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-08T07:11:25.343 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: Command line: ntpd -gq 2021-02-08T07:11:25.343 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: proto: precision = 0.127 usec (-23) 2021-02-08T07:11:25.344 INFO:teuthology.orchestra.run.smithi017.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-08T07:11:25.344 INFO:teuthology.orchestra.run.smithi017.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-08T07:11:25.344 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-08T07:11:25.345 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: restrict ::: KOD does nothing without LIMITED. 2021-02-08T07:11:25.345 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: Listen and drop on 0 v6wildcard [::]:123 2021-02-08T07:11:25.345 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-08T07:11:25.345 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: Listen normally on 2 lo 127.0.0.1:123 2021-02-08T07:11:25.345 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: Listen normally on 3 ens1f0 172.21.15.17:123 2021-02-08T07:11:25.345 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: Listen normally on 4 lo [::1]:123 2021-02-08T07:11:25.345 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe59:d992%3]:123 2021-02-08T07:11:25.345 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:25 ntpd[8253]: Listening on routing socket on fd #22 for interface updates 2021-02-08T07:11:26.343 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:26 ntpd[8253]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:26.343 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:26 ntpd[8253]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:26.343 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:26 ntpd[8253]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:26.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:26 ntpd[8253]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:28.343 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:28 ntpd[8253]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:28.343 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:28 ntpd[8253]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:28.343 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:28 ntpd[8253]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:28.343 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:28 ntpd[8253]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:30.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:30.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:30.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:30.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:30.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:30.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:30.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:30.345 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:30.345 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:30.345 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:30.345 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:30.345 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:30.345 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:30.345 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:30.345 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:30.345 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:30 ntpd[8253]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:32.343 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stderr: 8 Feb 07:11:32 ntpd[8253]: can't open /var/log/ntpstats/loopstats.20210208: Permission denied 2021-02-08T07:11:32.344 INFO:teuthology.orchestra.run.smithi017.stdout: 8 Feb 07:11:32 ntpd[8253]: ntpd: time set -0.028421 s 2021-02-08T07:11:32.345 INFO:teuthology.orchestra.run.smithi017.stdout:ntpd: time set -0.028421s 2021-02-08T07:11:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: remote refid st t when poll reach delay offset jitter 2021-02-08T07:11:32.394 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================== 2021-02-08T07:11:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-08T07:11:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-08T07:11:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-08T07:11:32.394 INFO:teuthology.orchestra.run.smithi017.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-08T07:11:32.395 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-08T07:11:32.395 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-08T07:11:32.448 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-08T07:11:32.448 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: Command line: ntpd -gq 2021-02-08T07:11:32.449 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: proto: precision = 0.123 usec (-23) 2021-02-08T07:11:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-08T07:11:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-08T07:11:32.450 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-08T07:11:32.450 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: restrict ::: KOD does nothing without LIMITED. 2021-02-08T07:11:32.450 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: Listen and drop on 0 v6wildcard [::]:123 2021-02-08T07:11:32.450 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-08T07:11:32.450 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: Listen normally on 2 lo 127.0.0.1:123 2021-02-08T07:11:32.450 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: Listen normally on 3 enp3s0f1 172.21.15.110:123 2021-02-08T07:11:32.450 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: Listen normally on 4 lo [::1]:123 2021-02-08T07:11:32.451 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc35%5]:123 2021-02-08T07:11:32.451 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:32 ntpd[8279]: Listening on routing socket on fd #22 for interface updates 2021-02-08T07:11:33.450 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:33.450 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:33.450 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:33.450 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:33.450 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:33.450 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:33.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:33 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:35.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:35.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:35.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:35.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:35.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:35.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:35.451 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:35.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:35.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:35.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:35.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:35.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:35.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:35.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:35.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:35.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:35 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:37.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:37.453 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:37.453 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:37.453 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:37.453 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:37 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:39.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:39 ntpd[8279]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-08T07:11:39.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:39 ntpd[8279]: can't open /var/log/ntpstats/rawstats.20210208: Permission denied 2021-02-08T07:11:39.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:39 ntpd[8279]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-08T07:11:39.452 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:39 ntpd[8279]: can't open /var/log/ntpstats/peerstats.20210208: Permission denied 2021-02-08T07:11:39.453 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:39 ntpd[8279]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-08T07:11:39.453 INFO:teuthology.orchestra.run.smithi110.stderr: 8 Feb 07:11:39 ntpd[8279]: can't open /var/log/ntpstats/loopstats.20210208: Permission denied 2021-02-08T07:11:39.453 INFO:teuthology.orchestra.run.smithi110.stdout: 8 Feb 07:11:39 ntpd[8279]: ntpd: time set -0.275671 s 2021-02-08T07:11:39.453 INFO:teuthology.orchestra.run.smithi110.stdout:ntpd: time set -0.275671s 2021-02-08T07:11:39.518 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2021-02-08T07:11:39.518 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2021-02-08T07:11:39.518 INFO:teuthology.orchestra.run.smithi110.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-08T07:11:39.518 INFO:teuthology.orchestra.run.smithi110.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-08T07:11:39.518 INFO:teuthology.orchestra.run.smithi110.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-08T07:11:39.518 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-08T07:11:39.519 INFO:teuthology.run_tasks:Running task install... 2021-02-08T07:11:39.531 DEBUG:teuthology.task.install:project ceph 2021-02-08T07:11:39.531 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '133532e25eccb46507b425783df68c050462a609'}} 2021-02-08T07:11:39.531 DEBUG:teuthology.task.install:config {'sha1': '133532e25eccb46507b425783df68c050462a609'} 2021-02-08T07:11:39.531 INFO:teuthology.task.install:Using flavor: basic 2021-02-08T07:11:39.548 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-02-08T07:11:39.548 INFO:teuthology.task.install:extra packages: [] 2021-02-08T07:11:39.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo lsb_release -is 2021-02-08T07:11:39.602 INFO:teuthology.orchestra.run.smithi017.stdout:Ubuntu 2021-02-08T07:11:39.607 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-08T07:11:39.607 DEBUG:teuthology.orchestra.run.smithi110:> sudo lsb_release -is 2021-02-08T07:11:39.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-key list | grep Ceph 2021-02-08T07:11:39.660 INFO:teuthology.orchestra.run.smithi110.stdout:Ubuntu 2021-02-08T07:11:39.665 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-08T07:11:39.665 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-key list | grep Ceph 2021-02-08T07:11:39.710 INFO:teuthology.orchestra.run.smithi017.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-08T07:11:39.753 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-08T07:11:39.767 INFO:teuthology.orchestra.run.smithi017.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-08T07:11:39.767 INFO:teuthology.orchestra.run.smithi017.stdout:uid [ unknown] Ceph.com (release key) 2021-02-08T07:11:39.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo lsb_release -is 2021-02-08T07:11:39.813 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-08T07:11:39.813 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph.com (release key) 2021-02-08T07:11:39.814 DEBUG:teuthology.orchestra.run.smithi110:> sudo lsb_release -is 2021-02-08T07:11:39.820 INFO:teuthology.orchestra.run.smithi017.stdout:Ubuntu 2021-02-08T07:11:39.825 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-08T07:11:39.825 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-08T07:11:39.825 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=133532e25eccb46507b425783df68c050462a609 2021-02-08T07:11:39.869 INFO:teuthology.orchestra.run.smithi110.stdout:Ubuntu 2021-02-08T07:11:39.870 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-08T07:11:39.870 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-08T07:11:39.870 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=133532e25eccb46507b425783df68c050462a609 2021-02-08T07:11:40.004 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default/ 2021-02-08T07:11:40.005 INFO:teuthology.task.install.deb:Package version is 15.2.8-294-g133532e2-1bionic 2021-02-08T07:11:40.028 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default/ 2021-02-08T07:11:40.028 INFO:teuthology.task.install.deb:Package version is 15.2.8-294-g133532e2-1bionic 2021-02-08T07:11:40.186 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:11:40.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-08T07:11:40.201 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get update 2021-02-08T07:11:40.206 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:11:40.207 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-08T07:11:40.222 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2021-02-08T07:11:40.398 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-08T07:11:40.413 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-08T07:11:40.499 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-08T07:11:40.521 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-08T07:11:40.577 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-08T07:11:40.580 INFO:teuthology.orchestra.run.smithi017.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic InRelease 2021-02-08T07:11:40.602 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-08T07:11:40.604 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic InRelease 2021-02-08T07:11:40.641 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-08T07:11:40.656 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-08T07:11:40.673 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-08T07:11:40.683 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-08T07:11:40.709 INFO:teuthology.orchestra.run.smithi017.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-08T07:11:40.740 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-08T07:11:40.777 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,081 B] 2021-02-08T07:11:40.811 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2021-02-08T07:11:40.869 INFO:teuthology.orchestra.run.smithi017.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2021-02-08T07:11:40.905 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,081 B] 2021-02-08T07:11:41.013 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 26.6 kB in 1s (44.6 kB/s) 2021-02-08T07:11:41.027 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 26.6 kB in 1s (44.2 kB/s) 2021-02-08T07:11:42.268 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2021-02-08T07:11:42.309 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.8-294-g133532e2-1bionic cephadm=15.2.8-294-g133532e2-1bionic ceph-mds=15.2.8-294-g133532e2-1bionic ceph-mgr=15.2.8-294-g133532e2-1bionic ceph-common=15.2.8-294-g133532e2-1bionic ceph-fuse=15.2.8-294-g133532e2-1bionic ceph-test=15.2.8-294-g133532e2-1bionic radosgw=15.2.8-294-g133532e2-1bionic python3-rados=15.2.8-294-g133532e2-1bionic python3-rgw=15.2.8-294-g133532e2-1bionic python3-cephfs=15.2.8-294-g133532e2-1bionic python3-rbd=15.2.8-294-g133532e2-1bionic libcephfs2=15.2.8-294-g133532e2-1bionic libcephfs-dev=15.2.8-294-g133532e2-1bionic librados2=15.2.8-294-g133532e2-1bionic librbd1=15.2.8-294-g133532e2-1bionic rbd-fuse=15.2.8-294-g133532e2-1bionic 2021-02-08T07:11:42.310 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2021-02-08T07:11:42.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.8-294-g133532e2-1bionic cephadm=15.2.8-294-g133532e2-1bionic ceph-mds=15.2.8-294-g133532e2-1bionic ceph-mgr=15.2.8-294-g133532e2-1bionic ceph-common=15.2.8-294-g133532e2-1bionic ceph-fuse=15.2.8-294-g133532e2-1bionic ceph-test=15.2.8-294-g133532e2-1bionic radosgw=15.2.8-294-g133532e2-1bionic python3-rados=15.2.8-294-g133532e2-1bionic python3-rgw=15.2.8-294-g133532e2-1bionic python3-cephfs=15.2.8-294-g133532e2-1bionic python3-rbd=15.2.8-294-g133532e2-1bionic libcephfs2=15.2.8-294-g133532e2-1bionic libcephfs-dev=15.2.8-294-g133532e2-1bionic librados2=15.2.8-294-g133532e2-1bionic librbd1=15.2.8-294-g133532e2-1bionic rbd-fuse=15.2.8-294-g133532e2-1bionic 2021-02-08T07:11:42.385 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2021-02-08T07:11:42.443 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2021-02-08T07:11:42.628 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2021-02-08T07:11:42.628 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2021-02-08T07:11:42.680 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2021-02-08T07:11:42.681 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2021-02-08T07:11:42.795 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2021-02-08T07:11:42.795 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-02-08T07:11:42.795 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-02-08T07:11:42.796 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2021-02-08T07:11:42.796 INFO:teuthology.orchestra.run.smithi110.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-08T07:11:42.796 INFO:teuthology.orchestra.run.smithi110.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-08T07:11:42.796 INFO:teuthology.orchestra.run.smithi110.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-08T07:11:42.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-08T07:11:42.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-08T07:11:42.798 INFO:teuthology.orchestra.run.smithi110.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-08T07:11:42.798 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-08T07:11:42.798 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-08T07:11:42.799 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2021-02-08T07:11:42.799 INFO:teuthology.orchestra.run.smithi110.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-08T07:11:42.799 INFO:teuthology.orchestra.run.smithi110.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-08T07:11:42.799 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-08T07:11:42.799 INFO:teuthology.orchestra.run.smithi110.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-08T07:11:42.799 INFO:teuthology.orchestra.run.smithi110.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-08T07:11:42.799 INFO:teuthology.orchestra.run.smithi110.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-08T07:11:42.799 INFO:teuthology.orchestra.run.smithi110.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-08T07:11:42.800 INFO:teuthology.orchestra.run.smithi110.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-08T07:11:42.800 INFO:teuthology.orchestra.run.smithi110.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-08T07:11:42.800 INFO:teuthology.orchestra.run.smithi110.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-08T07:11:42.800 INFO:teuthology.orchestra.run.smithi110.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-08T07:11:42.800 INFO:teuthology.orchestra.run.smithi110.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-08T07:11:42.800 INFO:teuthology.orchestra.run.smithi110.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-08T07:11:42.800 INFO:teuthology.orchestra.run.smithi110.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-08T07:11:42.800 INFO:teuthology.orchestra.run.smithi110.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-08T07:11:42.856 INFO:teuthology.orchestra.run.smithi017.stdout:The following additional packages will be installed: 2021-02-08T07:11:42.856 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-02-08T07:11:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-02-08T07:11:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2021-02-08T07:11:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-08T07:11:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-08T07:11:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-08T07:11:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-08T07:11:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-08T07:11:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-08T07:11:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-08T07:11:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-08T07:11:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-08T07:11:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-08T07:11:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout:Suggested packages: 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-08T07:11:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-08T07:11:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-08T07:11:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-08T07:11:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-08T07:11:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-08T07:11:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-08T07:11:42.901 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2021-02-08T07:11:42.902 INFO:teuthology.orchestra.run.smithi110.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-08T07:11:42.902 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-02-08T07:11:42.902 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-02-08T07:11:42.902 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-08T07:11:42.903 INFO:teuthology.orchestra.run.smithi110.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-08T07:11:42.904 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2021-02-08T07:11:42.905 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 librbd1 2021-02-08T07:11:42.968 INFO:teuthology.orchestra.run.smithi017.stdout:The following NEW packages will be installed: 2021-02-08T07:11:42.968 INFO:teuthology.orchestra.run.smithi017.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-08T07:11:42.968 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-02-08T07:11:42.968 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-02-08T07:11:42.969 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-02-08T07:11:42.969 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-08T07:11:42.969 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-08T07:11:42.969 INFO:teuthology.orchestra.run.smithi017.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-08T07:11:42.969 INFO:teuthology.orchestra.run.smithi017.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-08T07:11:42.969 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-08T07:11:42.970 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-08T07:11:42.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-08T07:11:42.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-08T07:11:42.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-08T07:11:42.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-08T07:11:42.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-08T07:11:42.972 INFO:teuthology.orchestra.run.smithi017.stdout:The following packages will be upgraded: 2021-02-08T07:11:42.972 INFO:teuthology.orchestra.run.smithi017.stdout: librados2 librbd1 2021-02-08T07:11:43.064 INFO:teuthology.orchestra.run.smithi110.stdout:2 upgraded, 111 newly installed, 0 to remove and 11 not upgraded. 2021-02-08T07:11:43.064 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 156 MB of archives. 2021-02-08T07:11:43.064 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 725 MB of additional disk space will be used. 2021-02-08T07:11:43.064 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.8-294-g133532e2-1bionic [1,637 kB] 2021-02-08T07:11:43.091 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-08T07:11:43.135 INFO:teuthology.orchestra.run.smithi017.stdout:2 upgraded, 111 newly installed, 0 to remove and 11 not upgraded. 2021-02-08T07:11:43.135 INFO:teuthology.orchestra.run.smithi017.stdout:Need to get 156 MB of archives. 2021-02-08T07:11:43.136 INFO:teuthology.orchestra.run.smithi017.stdout:After this operation, 725 MB of additional disk space will be used. 2021-02-08T07:11:43.136 INFO:teuthology.orchestra.run.smithi017.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.8-294-g133532e2-1bionic [1,637 kB] 2021-02-08T07:11:43.161 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-08T07:11:43.171 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-08T07:11:43.224 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.8-294-g133532e2-1bionic [3,061 kB] 2021-02-08T07:11:43.242 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-08T07:11:43.341 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.8-294-g133532e2-1bionic [435 kB] 2021-02-08T07:11:43.354 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.8-294-g133532e2-1bionic [332 kB] 2021-02-08T07:11:43.365 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.8-294-g133532e2-1bionic [20.7 kB] 2021-02-08T07:11:43.366 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.8-294-g133532e2-1bionic [133 kB] 2021-02-08T07:11:43.370 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.8-294-g133532e2-1bionic [29.0 kB] 2021-02-08T07:11:43.371 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.8-294-g133532e2-1bionic [312 kB] 2021-02-08T07:11:43.396 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.8-294-g133532e2-1bionic [2,902 kB] 2021-02-08T07:11:43.420 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-08T07:11:43.422 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-08T07:11:43.427 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-08T07:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-08T07:11:43.495 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.8-294-g133532e2-1bionic [99.2 kB] 2021-02-08T07:11:43.498 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.8-294-g133532e2-1bionic [339 kB] 2021-02-08T07:11:43.502 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-08T07:11:43.503 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.8-294-g133532e2-1bionic [3,061 kB] 2021-02-08T07:11:43.511 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-08T07:11:43.525 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.8-294-g133532e2-1bionic [18.8 MB] 2021-02-08T07:11:43.537 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-08T07:11:43.540 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-08T07:11:43.590 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-08T07:11:43.597 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-08T07:11:43.605 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-08T07:11:43.606 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-08T07:11:43.607 INFO:teuthology.orchestra.run.smithi017.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-08T07:11:43.615 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-08T07:11:43.623 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-08T07:11:43.644 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-08T07:11:43.648 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-08T07:11:43.657 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-08T07:11:43.663 INFO:teuthology.orchestra.run.smithi017.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-08T07:11:43.669 INFO:teuthology.orchestra.run.smithi017.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-08T07:11:43.670 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-08T07:11:43.676 INFO:teuthology.orchestra.run.smithi017.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-08T07:11:43.683 INFO:teuthology.orchestra.run.smithi017.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-08T07:11:43.689 INFO:teuthology.orchestra.run.smithi017.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-08T07:11:43.724 INFO:teuthology.orchestra.run.smithi017.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-08T07:11:43.727 INFO:teuthology.orchestra.run.smithi017.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-08T07:11:43.734 INFO:teuthology.orchestra.run.smithi017.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-08T07:11:43.736 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-08T07:11:43.737 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-08T07:11:43.738 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-08T07:11:43.740 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-08T07:11:43.740 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-08T07:11:43.741 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-08T07:11:43.743 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-08T07:11:43.746 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-08T07:11:43.747 INFO:teuthology.orchestra.run.smithi017.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-08T07:11:43.814 INFO:teuthology.orchestra.run.smithi017.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-08T07:11:43.814 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-08T07:11:43.815 INFO:teuthology.orchestra.run.smithi017.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-08T07:11:43.815 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-08T07:11:43.815 INFO:teuthology.orchestra.run.smithi017.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-08T07:11:43.818 INFO:teuthology.orchestra.run.smithi017.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-08T07:11:43.818 INFO:teuthology.orchestra.run.smithi017.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-08T07:11:43.819 INFO:teuthology.orchestra.run.smithi017.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-08T07:11:43.822 INFO:teuthology.orchestra.run.smithi017.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-08T07:11:43.824 INFO:teuthology.orchestra.run.smithi017.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-08T07:11:43.893 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-08T07:11:43.894 INFO:teuthology.orchestra.run.smithi017.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-08T07:11:43.894 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-08T07:11:43.894 INFO:teuthology.orchestra.run.smithi017.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-08T07:11:43.895 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-08T07:11:43.895 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-08T07:11:43.898 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-08T07:11:43.905 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-08T07:11:43.909 INFO:teuthology.orchestra.run.smithi110.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-08T07:11:43.910 INFO:teuthology.orchestra.run.smithi110.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-08T07:11:43.922 INFO:teuthology.orchestra.run.smithi017.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.8-294-g133532e2-1bionic [435 kB] 2021-02-08T07:11:43.971 INFO:teuthology.orchestra.run.smithi110.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-08T07:11:43.972 INFO:teuthology.orchestra.run.smithi017.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-08T07:11:43.973 INFO:teuthology.orchestra.run.smithi017.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-08T07:11:43.974 INFO:teuthology.orchestra.run.smithi017.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-08T07:11:43.976 INFO:teuthology.orchestra.run.smithi017.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-08T07:11:43.978 INFO:teuthology.orchestra.run.smithi017.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-08T07:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.8-294-g133532e2-1bionic [332 kB] 2021-02-08T07:11:43.985 INFO:teuthology.orchestra.run.smithi017.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-08T07:11:43.989 INFO:teuthology.orchestra.run.smithi017.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-08T07:11:43.990 INFO:teuthology.orchestra.run.smithi017.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-08T07:11:44.046 INFO:teuthology.orchestra.run.smithi017.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.8-294-g133532e2-1bionic [20.7 kB] 2021-02-08T07:11:44.047 INFO:teuthology.orchestra.run.smithi017.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.8-294-g133532e2-1bionic [133 kB] 2021-02-08T07:11:44.050 INFO:teuthology.orchestra.run.smithi110.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-08T07:11:44.050 INFO:teuthology.orchestra.run.smithi110.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-08T07:11:44.052 INFO:teuthology.orchestra.run.smithi017.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-08T07:11:44.102 INFO:teuthology.orchestra.run.smithi017.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.8-294-g133532e2-1bionic [29.0 kB] 2021-02-08T07:11:44.103 INFO:teuthology.orchestra.run.smithi017.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.8-294-g133532e2-1bionic [312 kB] 2021-02-08T07:11:44.116 INFO:teuthology.orchestra.run.smithi017.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.8-294-g133532e2-1bionic [2,902 kB] 2021-02-08T07:11:44.134 INFO:teuthology.orchestra.run.smithi017.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-08T07:11:44.135 INFO:teuthology.orchestra.run.smithi017.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-08T07:11:44.328 INFO:teuthology.orchestra.run.smithi110.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-08T07:11:44.372 INFO:teuthology.orchestra.run.smithi110.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-08T07:11:44.374 INFO:teuthology.orchestra.run.smithi110.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-08T07:11:44.401 INFO:teuthology.orchestra.run.smithi110.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.8-294-g133532e2-1bionic [5,172 kB] 2021-02-08T07:11:44.406 INFO:teuthology.orchestra.run.smithi110.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-08T07:11:44.407 INFO:teuthology.orchestra.run.smithi110.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-08T07:11:44.407 INFO:teuthology.orchestra.run.smithi110.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-08T07:11:44.409 INFO:teuthology.orchestra.run.smithi110.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-08T07:11:44.410 INFO:teuthology.orchestra.run.smithi110.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-08T07:11:44.411 INFO:teuthology.orchestra.run.smithi110.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-08T07:11:44.411 INFO:teuthology.orchestra.run.smithi110.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-08T07:11:44.417 INFO:teuthology.orchestra.run.smithi017.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-08T07:11:44.433 INFO:teuthology.orchestra.run.smithi017.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-08T07:11:44.457 INFO:teuthology.orchestra.run.smithi017.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-08T07:11:44.485 INFO:teuthology.orchestra.run.smithi110.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-08T07:11:44.486 INFO:teuthology.orchestra.run.smithi110.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-08T07:11:44.503 INFO:teuthology.orchestra.run.smithi017.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-08T07:11:44.504 INFO:teuthology.orchestra.run.smithi017.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-08T07:11:44.504 INFO:teuthology.orchestra.run.smithi017.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-08T07:11:44.504 INFO:teuthology.orchestra.run.smithi017.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-08T07:11:44.505 INFO:teuthology.orchestra.run.smithi017.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-08T07:11:44.505 INFO:teuthology.orchestra.run.smithi017.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-08T07:11:44.506 INFO:teuthology.orchestra.run.smithi017.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-08T07:11:44.564 INFO:teuthology.orchestra.run.smithi110.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-08T07:11:44.565 INFO:teuthology.orchestra.run.smithi110.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-08T07:11:44.570 INFO:teuthology.orchestra.run.smithi110.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-08T07:11:44.571 INFO:teuthology.orchestra.run.smithi110.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-08T07:11:44.572 INFO:teuthology.orchestra.run.smithi110.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-08T07:11:44.575 INFO:teuthology.orchestra.run.smithi110.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-08T07:11:44.577 INFO:teuthology.orchestra.run.smithi017.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-08T07:11:44.578 INFO:teuthology.orchestra.run.smithi110.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-08T07:11:44.578 INFO:teuthology.orchestra.run.smithi110.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-08T07:11:44.579 INFO:teuthology.orchestra.run.smithi017.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-08T07:11:44.613 INFO:teuthology.orchestra.run.smithi017.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.8-294-g133532e2-1bionic [99.2 kB] 2021-02-08T07:11:44.642 INFO:teuthology.orchestra.run.smithi110.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-08T07:11:44.649 INFO:teuthology.orchestra.run.smithi110.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-08T07:11:44.652 INFO:teuthology.orchestra.run.smithi110.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.8-294-g133532e2-1bionic [152 kB] 2021-02-08T07:11:44.656 INFO:teuthology.orchestra.run.smithi017.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-08T07:11:44.657 INFO:teuthology.orchestra.run.smithi017.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-08T07:11:44.662 INFO:teuthology.orchestra.run.smithi110.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.8-294-g133532e2-1bionic [1,284 kB] 2021-02-08T07:11:44.663 INFO:teuthology.orchestra.run.smithi017.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-08T07:11:44.663 INFO:teuthology.orchestra.run.smithi017.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-08T07:11:44.664 INFO:teuthology.orchestra.run.smithi017.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-08T07:11:44.667 INFO:teuthology.orchestra.run.smithi017.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-08T07:11:44.670 INFO:teuthology.orchestra.run.smithi017.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-08T07:11:44.672 INFO:teuthology.orchestra.run.smithi017.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-08T07:11:44.673 INFO:teuthology.orchestra.run.smithi017.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.8-294-g133532e2-1bionic [339 kB] 2021-02-08T07:11:44.721 INFO:teuthology.orchestra.run.smithi110.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-08T07:11:44.722 INFO:teuthology.orchestra.run.smithi110.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.8-294-g133532e2-1bionic [5,944 kB] 2021-02-08T07:11:44.727 INFO:teuthology.orchestra.run.smithi110.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-08T07:11:44.728 INFO:teuthology.orchestra.run.smithi110.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-08T07:11:44.729 INFO:teuthology.orchestra.run.smithi110.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-08T07:11:44.729 INFO:teuthology.orchestra.run.smithi110.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-08T07:11:44.731 INFO:teuthology.orchestra.run.smithi110.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-08T07:11:44.734 INFO:teuthology.orchestra.run.smithi110.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-08T07:11:44.734 INFO:teuthology.orchestra.run.smithi110.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-08T07:11:44.735 INFO:teuthology.orchestra.run.smithi017.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-08T07:11:44.736 INFO:teuthology.orchestra.run.smithi017.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.8-294-g133532e2-1bionic [18.8 MB] 2021-02-08T07:11:44.742 INFO:teuthology.orchestra.run.smithi017.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-08T07:11:44.800 INFO:teuthology.orchestra.run.smithi110.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-08T07:11:44.800 INFO:teuthology.orchestra.run.smithi110.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-08T07:11:44.815 INFO:teuthology.orchestra.run.smithi017.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-08T07:11:44.822 INFO:teuthology.orchestra.run.smithi017.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-08T07:11:44.823 INFO:teuthology.orchestra.run.smithi017.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-08T07:11:44.824 INFO:teuthology.orchestra.run.smithi017.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-08T07:11:44.824 INFO:teuthology.orchestra.run.smithi017.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-08T07:11:44.826 INFO:teuthology.orchestra.run.smithi017.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-08T07:11:44.828 INFO:teuthology.orchestra.run.smithi017.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-08T07:11:44.829 INFO:teuthology.orchestra.run.smithi017.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-08T07:11:44.878 INFO:teuthology.orchestra.run.smithi110.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-08T07:11:44.894 INFO:teuthology.orchestra.run.smithi017.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-08T07:11:44.895 INFO:teuthology.orchestra.run.smithi017.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-08T07:11:44.896 INFO:teuthology.orchestra.run.smithi110.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-08T07:11:44.896 INFO:teuthology.orchestra.run.smithi110.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-08T07:11:44.897 INFO:teuthology.orchestra.run.smithi110.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-08T07:11:44.974 INFO:teuthology.orchestra.run.smithi017.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-08T07:11:44.984 INFO:teuthology.orchestra.run.smithi110.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-08T07:11:44.984 INFO:teuthology.orchestra.run.smithi110.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-08T07:11:44.985 INFO:teuthology.orchestra.run.smithi110.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-08T07:11:44.998 INFO:teuthology.orchestra.run.smithi017.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-08T07:11:44.998 INFO:teuthology.orchestra.run.smithi017.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-08T07:11:44.999 INFO:teuthology.orchestra.run.smithi017.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-08T07:11:45.015 INFO:teuthology.orchestra.run.smithi110.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.8-294-g133532e2-1bionic [22.8 MB] 2021-02-08T07:11:45.038 INFO:teuthology.orchestra.run.smithi110.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-08T07:11:45.039 INFO:teuthology.orchestra.run.smithi110.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-08T07:11:45.040 INFO:teuthology.orchestra.run.smithi110.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-08T07:11:45.041 INFO:teuthology.orchestra.run.smithi110.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-08T07:11:45.045 INFO:teuthology.orchestra.run.smithi110.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-08T07:11:45.045 INFO:teuthology.orchestra.run.smithi110.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-08T07:11:45.136 INFO:teuthology.orchestra.run.smithi017.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-08T07:11:45.136 INFO:teuthology.orchestra.run.smithi017.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-08T07:11:45.138 INFO:teuthology.orchestra.run.smithi017.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-08T07:11:45.146 INFO:teuthology.orchestra.run.smithi017.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-08T07:11:45.146 INFO:teuthology.orchestra.run.smithi017.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-08T07:11:45.148 INFO:teuthology.orchestra.run.smithi017.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-08T07:11:45.148 INFO:teuthology.orchestra.run.smithi017.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-08T07:11:45.149 INFO:teuthology.orchestra.run.smithi017.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-08T07:11:45.150 INFO:teuthology.orchestra.run.smithi017.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-08T07:11:46.279 INFO:teuthology.orchestra.run.smithi110.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.8-294-g133532e2-1bionic [3,920 B] 2021-02-08T07:11:46.280 INFO:teuthology.orchestra.run.smithi110.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.8-294-g133532e2-1bionic [515 kB] 2021-02-08T07:11:46.324 INFO:teuthology.orchestra.run.smithi110.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.8-294-g133532e2-1bionic [1,854 kB] 2021-02-08T07:11:46.415 INFO:teuthology.orchestra.run.smithi110.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.8-294-g133532e2-1bionic [52.2 kB] 2021-02-08T07:11:46.417 INFO:teuthology.orchestra.run.smithi110.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.8-294-g133532e2-1bionic [57.7 kB] 2021-02-08T07:11:46.419 INFO:teuthology.orchestra.run.smithi110.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.8-294-g133532e2-1bionic [3,200 kB] 2021-02-08T07:11:46.620 INFO:teuthology.orchestra.run.smithi110.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.8-294-g133532e2-1bionic [34.3 kB] 2021-02-08T07:11:46.621 INFO:teuthology.orchestra.run.smithi110.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.8-294-g133532e2-1bionic [7,886 kB] 2021-02-08T07:11:47.131 INFO:teuthology.orchestra.run.smithi110.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.8-294-g133532e2-1bionic [4,036 B] 2021-02-08T07:11:50.955 INFO:teuthology.orchestra.run.smithi017.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.8-294-g133532e2-1bionic [5,172 kB] 2021-02-08T07:11:50.955 INFO:teuthology.orchestra.run.smithi017.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.8-294-g133532e2-1bionic [152 kB] 2021-02-08T07:11:50.955 INFO:teuthology.orchestra.run.smithi017.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.8-294-g133532e2-1bionic [1,284 kB] 2021-02-08T07:11:50.956 INFO:teuthology.orchestra.run.smithi017.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.8-294-g133532e2-1bionic [5,944 kB] 2021-02-08T07:11:50.956 INFO:teuthology.orchestra.run.smithi017.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.8-294-g133532e2-1bionic [22.8 MB] 2021-02-08T07:11:50.956 INFO:teuthology.orchestra.run.smithi110.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.8-294-g133532e2-1bionic [22.5 kB] 2021-02-08T07:11:50.956 INFO:teuthology.orchestra.run.smithi110.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.8-294-g133532e2-1bionic [44.0 MB] 2021-02-08T07:11:50.956 INFO:teuthology.orchestra.run.smithi110.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.8-294-g133532e2-1bionic [18.0 kB] 2021-02-08T07:11:50.956 INFO:teuthology.orchestra.run.smithi110.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.8-294-g133532e2-1bionic [8,878 kB] 2021-02-08T07:11:50.956 INFO:teuthology.orchestra.run.smithi110.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.8-294-g133532e2-1bionic [63.2 kB] 2021-02-08T07:11:51.034 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 156 MB in 7s (21.1 MB/s) 2021-02-08T07:11:51.176 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-08T07:11:51.223 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-08T07:11:51.227 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-08T07:11:51.266 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-08T07:11:51.345 INFO:teuthology.orchestra.run.smithi017.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.8-294-g133532e2-1bionic [3,920 B] 2021-02-08T07:11:51.346 INFO:teuthology.orchestra.run.smithi017.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.8-294-g133532e2-1bionic [515 kB] 2021-02-08T07:11:51.372 INFO:teuthology.orchestra.run.smithi017.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.8-294-g133532e2-1bionic [1,854 kB] 2021-02-08T07:11:51.465 INFO:teuthology.orchestra.run.smithi017.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.8-294-g133532e2-1bionic [52.2 kB] 2021-02-08T07:11:51.466 INFO:teuthology.orchestra.run.smithi017.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.8-294-g133532e2-1bionic [57.7 kB] 2021-02-08T07:11:51.469 INFO:teuthology.orchestra.run.smithi017.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.8-294-g133532e2-1bionic [3,200 kB] 2021-02-08T07:11:51.578 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-08T07:11:51.591 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-08T07:11:51.634 INFO:teuthology.orchestra.run.smithi017.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.8-294-g133532e2-1bionic [34.3 kB] 2021-02-08T07:11:51.635 INFO:teuthology.orchestra.run.smithi017.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.8-294-g133532e2-1bionic [7,886 kB] 2021-02-08T07:11:51.670 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-08T07:11:52.038 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../002-librbd1_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:52.039 INFO:teuthology.orchestra.run.smithi017.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.8-294-g133532e2-1bionic [4,036 B] 2021-02-08T07:11:52.039 INFO:teuthology.orchestra.run.smithi017.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.8-294-g133532e2-1bionic [22.5 kB] 2021-02-08T07:11:52.040 INFO:teuthology.orchestra.run.smithi017.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.8-294-g133532e2-1bionic [44.0 MB] 2021-02-08T07:11:52.165 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librbd1 (15.2.8-294-g133532e2-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-08T07:11:52.718 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../003-librados2_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:52.845 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librados2 (15.2.8-294-g133532e2-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-08T07:11:53.516 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs2. 2021-02-08T07:11:53.528 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../004-libcephfs2_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:53.566 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:53.851 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rados. 2021-02-08T07:11:53.866 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../005-python3-rados_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:53.918 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rados (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:54.152 INFO:teuthology.orchestra.run.smithi017.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.8-294-g133532e2-1bionic [18.0 kB] 2021-02-08T07:11:54.152 INFO:teuthology.orchestra.run.smithi017.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.8-294-g133532e2-1bionic [8,878 kB] 2021-02-08T07:11:54.263 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-08T07:11:54.277 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:11:54.321 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-argparse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:54.544 INFO:teuthology.orchestra.run.smithi017.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2021-02-03-1046-octopus/133532e25eccb46507b425783df68c050462a609/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.8-294-g133532e2-1bionic [63.2 kB] 2021-02-08T07:11:54.624 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cephfs. 2021-02-08T07:11:54.638 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../007-python3-cephfs_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:54.682 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cephfs (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:55.002 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-08T07:11:55.018 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:11:55.060 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-common (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:55.276 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 156 MB in 12s (13.5 MB/s) 2021-02-08T07:11:55.363 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-prettytable. 2021-02-08T07:11:55.376 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2021-02-08T07:11:55.399 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-08T07:11:55.413 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-08T07:11:55.448 INFO:teuthology.orchestra.run.smithi017.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-08T07:11:55.452 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-08T07:11:55.489 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-08T07:11:55.649 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rbd. 2021-02-08T07:11:55.665 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../010-python3-rbd_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:55.707 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rbd (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:55.750 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-08T07:11:55.764 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-08T07:11:55.842 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-08T07:11:56.077 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-08T07:11:56.090 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-08T07:11:56.135 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-08T07:11:56.217 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../002-librbd1_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:56.346 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librbd1 (15.2.8-294-g133532e2-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-08T07:11:56.455 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-08T07:11:56.469 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-08T07:11:56.514 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-08T07:11:56.850 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librgw2. 2021-02-08T07:11:56.857 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../003-librados2_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:56.864 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../013-librgw2_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:56.916 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librgw2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:56.966 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librados2 (15.2.8-294-g133532e2-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-08T07:11:57.427 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rgw. 2021-02-08T07:11:57.441 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../014-python3-rgw_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:57.486 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rgw (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:57.611 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libcephfs2. 2021-02-08T07:11:57.624 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../004-libcephfs2_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:57.661 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libcephfs2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:57.788 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-08T07:11:57.802 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-08T07:11:57.839 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-08T07:11:57.939 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rados. 2021-02-08T07:11:57.952 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../005-python3-rados_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:57.989 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rados (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:58.150 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-08T07:11:58.164 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-08T07:11:58.208 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-08T07:11:58.291 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-08T07:11:58.305 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:11:58.333 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-ceph-argparse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:58.594 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cephfs. 2021-02-08T07:11:58.595 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liboath0. 2021-02-08T07:11:58.609 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2021-02-08T07:11:58.610 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../007-python3-cephfs_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:58.644 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cephfs (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:58.654 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-08T07:11:58.939 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libradosstriper1. 2021-02-08T07:11:58.955 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../018-libradosstriper1_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:58.972 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-08T07:11:58.987 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:11:58.997 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libradosstriper1 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:59.030 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-ceph-common (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:59.524 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-prettytable. 2021-02-08T07:11:59.541 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2021-02-08T07:11:59.583 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-08T07:11:59.651 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-common. 2021-02-08T07:11:59.665 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../019-ceph-common_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:59.709 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-common (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:11:59.810 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rbd. 2021-02-08T07:11:59.823 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../010-python3-rbd_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:11:59.852 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rbd (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:00.154 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-08T07:12:00.168 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-08T07:12:00.204 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-08T07:12:00.532 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-08T07:12:00.545 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-08T07:12:00.582 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-08T07:12:00.910 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package librgw2. 2021-02-08T07:12:00.923 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../013-librgw2_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:00.985 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librgw2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:01.457 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-base. 2021-02-08T07:12:01.471 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../020-ceph-base_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:01.478 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rgw. 2021-02-08T07:12:01.492 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../014-python3-rgw_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:01.520 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rgw (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:01.524 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-base (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:01.807 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-08T07:12:01.820 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-08T07:12:01.864 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-08T07:12:02.167 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-08T07:12:02.183 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-08T07:12:02.242 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-08T07:12:02.412 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-dateutil. 2021-02-08T07:12:02.426 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2021-02-08T07:12:02.470 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-08T07:12:02.578 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package liboath0. 2021-02-08T07:12:02.592 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2021-02-08T07:12:02.628 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-08T07:12:02.831 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-08T07:12:02.847 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-08T07:12:02.889 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libradosstriper1. 2021-02-08T07:12:02.890 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-08T07:12:02.904 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../018-libradosstriper1_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:02.939 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libradosstriper1 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:03.184 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-08T07:12:03.197 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-08T07:12:03.234 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-08T07:12:03.384 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-common. 2021-02-08T07:12:03.397 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../019-ceph-common_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:03.459 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-common (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:03.537 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cryptography. 2021-02-08T07:12:03.550 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-08T07:12:03.587 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-08T07:12:03.983 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-openssl. 2021-02-08T07:12:03.997 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-08T07:12:04.041 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-08T07:12:04.302 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-08T07:12:04.318 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:04.360 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-modules-core (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:04.715 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-08T07:12:04.729 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-08T07:12:04.773 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-08T07:12:05.076 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-08T07:12:05.090 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-08T07:12:05.134 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-08T07:12:05.155 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-base. 2021-02-08T07:12:05.171 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../020-ceph-base_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:05.215 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-base (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:05.505 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jwt. 2021-02-08T07:12:05.518 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-08T07:12:05.555 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-08T07:12:05.891 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-logutils. 2021-02-08T07:12:05.905 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2021-02-08T07:12:05.950 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-08T07:12:06.110 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-dateutil. 2021-02-08T07:12:06.124 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2021-02-08T07:12:06.169 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-08T07:12:06.244 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-08T07:12:06.258 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-08T07:12:06.303 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-08T07:12:06.471 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-08T07:12:06.486 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-08T07:12:06.521 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-08T07:12:06.597 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-mako. 2021-02-08T07:12:06.611 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-08T07:12:06.656 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-08T07:12:06.808 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-08T07:12:06.822 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-08T07:12:06.866 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-08T07:12:06.975 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-08T07:12:06.991 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-08T07:12:07.034 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-08T07:12:07.210 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cryptography. 2021-02-08T07:12:07.226 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-08T07:12:07.293 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-08T07:12:07.370 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-08T07:12:07.384 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-08T07:12:07.437 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-08T07:12:07.680 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-openssl. 2021-02-08T07:12:07.693 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-08T07:12:07.738 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-08T07:12:07.739 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webob. 2021-02-08T07:12:07.754 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-08T07:12:07.798 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-08T07:12:07.999 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-08T07:12:08.013 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:08.057 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-modules-core (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:08.117 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-bs4. 2021-02-08T07:12:08.133 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2021-02-08T07:12:08.176 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-08T07:12:08.428 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-08T07:12:08.445 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-08T07:12:08.483 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-waitress. 2021-02-08T07:12:08.486 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-08T07:12:08.498 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2021-02-08T07:12:08.542 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-08T07:12:08.831 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-08T07:12:08.845 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-08T07:12:08.854 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tempita. 2021-02-08T07:12:08.870 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2021-02-08T07:12:08.889 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-08T07:12:08.912 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-08T07:12:09.217 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-jwt. 2021-02-08T07:12:09.224 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-paste. 2021-02-08T07:12:09.233 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-08T07:12:09.240 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-08T07:12:09.275 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-08T07:12:09.283 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-08T07:12:09.586 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-08T07:12:09.602 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-08T07:12:09.628 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-logutils. 2021-02-08T07:12:09.643 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2021-02-08T07:12:09.644 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-08T07:12:09.695 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-08T07:12:09.941 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-08T07:12:09.956 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-08T07:12:09.999 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-08T07:12:10.014 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-08T07:12:10.029 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-08T07:12:10.081 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-08T07:12:10.293 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webtest. 2021-02-08T07:12:10.307 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-08T07:12:10.343 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-08T07:12:10.403 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-mako. 2021-02-08T07:12:10.416 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-08T07:12:10.461 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-08T07:12:10.713 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pecan. 2021-02-08T07:12:10.726 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2021-02-08T07:12:10.763 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-08T07:12:10.789 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-08T07:12:10.804 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-08T07:12:10.855 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-08T07:12:11.118 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-08T07:12:11.132 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-08T07:12:11.158 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-08T07:12:11.174 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-08T07:12:11.176 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-08T07:12:11.216 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-08T07:12:11.536 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-webob. 2021-02-08T07:12:11.546 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr. 2021-02-08T07:12:11.549 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-08T07:12:11.561 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../045-ceph-mgr_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:11.586 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-08T07:12:11.604 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:11.897 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-bs4. 2021-02-08T07:12:11.912 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2021-02-08T07:12:11.955 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-08T07:12:12.074 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mon. 2021-02-08T07:12:12.089 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../046-ceph-mon_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:12.133 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mon (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:12.310 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-waitress. 2021-02-08T07:12:12.326 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2021-02-08T07:12:12.368 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-08T07:12:12.689 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-tempita. 2021-02-08T07:12:12.702 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2021-02-08T07:12:12.739 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-08T07:12:13.003 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-osd. 2021-02-08T07:12:13.015 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../047-ceph-osd_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:13.053 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-osd (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:13.067 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-paste. 2021-02-08T07:12:13.080 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-08T07:12:13.125 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-08T07:12:13.445 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-08T07:12:13.460 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-08T07:12:13.503 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-08T07:12:13.864 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-08T07:12:13.879 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-08T07:12:13.914 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-08T07:12:14.217 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-webtest. 2021-02-08T07:12:14.231 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-08T07:12:14.275 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-08T07:12:15.312 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pecan. 2021-02-08T07:12:15.327 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2021-02-08T07:12:15.370 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-08T07:12:15.708 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-08T07:12:15.723 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-08T07:12:15.766 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-08T07:12:15.975 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph. 2021-02-08T07:12:15.991 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../048-ceph_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:16.034 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:16.144 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr. 2021-02-08T07:12:16.161 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../045-ceph-mgr_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:16.202 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:16.337 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-fuse. 2021-02-08T07:12:16.352 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../049-ceph-fuse_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:16.396 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-fuse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:16.664 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mon. 2021-02-08T07:12:16.680 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../046-ceph-mon_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:16.722 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mon (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:16.817 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mds. 2021-02-08T07:12:16.830 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../050-ceph-mds_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:16.866 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mds (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:17.369 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephadm. 2021-02-08T07:12:17.384 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../051-cephadm_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:17.428 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephadm (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:17.642 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-osd. 2021-02-08T07:12:17.753 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../047-ceph-osd_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:17.753 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-osd (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:17.756 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jinja2. 2021-02-08T07:12:17.770 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-08T07:12:17.814 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-08T07:12:18.134 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-08T07:12:18.149 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:18.193 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-cephadm (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:18.554 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-08T07:12:18.569 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2021-02-08T07:12:18.621 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-08T07:12:18.907 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-routes. 2021-02-08T07:12:18.922 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2021-02-08T07:12:18.992 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-08T07:12:19.304 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-08T07:12:19.318 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:19.362 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-dashboard (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:19.714 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph. 2021-02-08T07:12:19.730 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../048-ceph_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:19.773 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:20.127 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-fuse. 2021-02-08T07:12:20.144 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../049-ceph-fuse_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:20.185 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-fuse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:20.243 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2021-02-08T07:12:20.260 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:20.301 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:20.572 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mds. 2021-02-08T07:12:20.585 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../050-ceph-mds_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:20.613 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-numpy. 2021-02-08T07:12:20.626 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-08T07:12:20.630 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mds (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:20.695 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-08T07:12:21.150 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package cephadm. 2021-02-08T07:12:21.164 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../051-cephadm_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:21.225 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking cephadm (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:21.303 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-decorator. 2021-02-08T07:12:21.318 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-08T07:12:21.361 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-08T07:12:21.544 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-jinja2. 2021-02-08T07:12:21.559 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-08T07:12:21.611 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-08T07:12:21.697 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-scipy. 2021-02-08T07:12:21.712 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-08T07:12:21.755 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-08T07:12:21.939 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-08T07:12:21.953 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:21.997 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-cephadm (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:22.325 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-08T07:12:22.340 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2021-02-08T07:12:22.384 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-08T07:12:22.711 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-routes. 2021-02-08T07:12:22.727 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2021-02-08T07:12:22.778 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-08T07:12:23.117 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-08T07:12:23.123 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-08T07:12:23.133 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:23.137 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-08T07:12:23.175 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-dashboard (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:23.181 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-08T07:12:23.635 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-joblib. 2021-02-08T07:12:23.649 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-08T07:12:23.693 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-08T07:12:24.040 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn. 2021-02-08T07:12:24.054 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-08T07:12:24.115 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-08T07:12:24.122 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2021-02-08T07:12:24.139 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:24.180 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:24.508 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-numpy. 2021-02-08T07:12:24.523 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-08T07:12:24.566 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-08T07:12:24.652 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-08T07:12:24.666 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:24.710 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:25.180 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-decorator. 2021-02-08T07:12:25.197 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-08T07:12:25.239 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-08T07:12:25.549 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-scipy. 2021-02-08T07:12:25.563 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-08T07:12:25.608 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-08T07:12:25.797 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-monotonic. 2021-02-08T07:12:25.810 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-08T07:12:25.855 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-08T07:12:26.085 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-fasteners. 2021-02-08T07:12:26.101 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-08T07:12:26.135 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-08T07:12:26.393 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-httplib2. 2021-02-08T07:12:26.401 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-08T07:12:26.435 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-08T07:12:26.729 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-08T07:12:26.743 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-08T07:12:26.788 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-08T07:12:26.850 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-08T07:12:26.863 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-08T07:12:26.908 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-08T07:12:27.116 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-08T07:12:27.131 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-08T07:12:27.174 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-08T07:12:27.344 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-joblib. 2021-02-08T07:12:27.361 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-08T07:12:27.403 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-08T07:12:27.519 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rsa. 2021-02-08T07:12:27.534 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-08T07:12:27.569 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-08T07:12:27.766 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-sklearn. 2021-02-08T07:12:27.780 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-08T07:12:27.824 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-08T07:12:27.830 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-08T07:12:27.845 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-08T07:12:27.880 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-08T07:12:28.293 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-08T07:12:28.308 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-08T07:12:28.343 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-08T07:12:28.379 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-08T07:12:28.393 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:28.437 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:28.654 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-setuptools. 2021-02-08T07:12:28.670 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-08T07:12:28.713 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-08T07:12:29.091 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-websocket. 2021-02-08T07:12:29.105 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-08T07:12:29.177 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-08T07:12:29.478 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-08T07:12:29.490 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-monotonic. 2021-02-08T07:12:29.494 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-08T07:12:29.505 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-08T07:12:29.536 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-08T07:12:29.540 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-08T07:12:29.845 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-fasteners. 2021-02-08T07:12:29.860 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-08T07:12:29.904 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-08T07:12:30.357 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-08T07:12:30.370 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:30.440 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-k8sevents (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:30.694 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-httplib2. 2021-02-08T07:12:30.709 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-08T07:12:30.768 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-08T07:12:30.769 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-08T07:12:30.782 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-08T07:12:30.826 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-08T07:12:31.080 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-08T07:12:31.096 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-08T07:12:31.139 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-08T07:12:31.196 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-08T07:12:31.210 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-08T07:12:31.257 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-08T07:12:31.500 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-08T07:12:31.516 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-08T07:12:31.559 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-08T07:12:31.565 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-08T07:12:31.581 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:31.624 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-rook (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:31.853 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rsa. 2021-02-08T07:12:31.870 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-08T07:12:31.911 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-08T07:12:31.935 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-08T07:12:31.951 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-08T07:12:32.073 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-08T07:12:32.223 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-08T07:12:32.237 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-08T07:12:32.281 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-08T07:12:32.434 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-08T07:12:32.449 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-08T07:12:32.501 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-08T07:12:32.804 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package jq. 2021-02-08T07:12:32.819 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-08T07:12:32.820 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-08T07:12:32.833 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-08T07:12:32.862 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-08T07:12:32.877 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-08T07:12:33.123 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package socat. 2021-02-08T07:12:33.137 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-08T07:12:33.181 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-08T07:12:33.196 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-setuptools. 2021-02-08T07:12:33.213 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-08T07:12:33.255 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-08T07:12:33.602 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-08T07:12:33.617 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-08T07:12:33.660 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-08T07:12:33.666 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-websocket. 2021-02-08T07:12:33.683 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-08T07:12:33.725 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-08T07:12:33.938 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package xmlstarlet. 2021-02-08T07:12:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-08T07:12:33.988 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-08T07:12:34.061 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-08T07:12:34.075 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-08T07:12:34.119 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-08T07:12:34.350 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-test. 2021-02-08T07:12:34.364 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../086-ceph-test_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:34.408 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-test (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:34.531 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-08T07:12:34.545 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:34.589 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-k8sevents (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:34.909 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-08T07:12:34.924 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-08T07:12:34.967 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-08T07:12:35.362 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-08T07:12:35.376 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-08T07:12:35.423 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-08T07:12:35.740 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-08T07:12:35.756 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.8-294-g133532e2-1bionic_all.deb ... 2021-02-08T07:12:35.798 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-rook (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:36.109 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-08T07:12:36.124 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-08T07:12:36.255 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-08T07:12:36.650 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-08T07:12:36.664 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-08T07:12:36.708 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-08T07:12:36.977 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package jq. 2021-02-08T07:12:36.991 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-08T07:12:37.035 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-08T07:12:37.321 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package socat. 2021-02-08T07:12:37.338 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-08T07:12:37.380 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-08T07:12:37.779 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-08T07:12:37.793 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:37.800 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-08T07:12:37.816 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-08T07:12:37.837 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs-dev (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:37.858 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-08T07:12:38.124 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package nvme-cli. 2021-02-08T07:12:38.138 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-08T07:12:38.144 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package xmlstarlet. 2021-02-08T07:12:38.160 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-08T07:12:38.182 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-08T07:12:38.202 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-08T07:12:38.537 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-attr. 2021-02-08T07:12:38.540 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-test. 2021-02-08T07:12:38.552 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-08T07:12:38.557 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../086-ceph-test_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:38.595 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-08T07:12:38.598 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-test (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:38.898 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-funcsigs. 2021-02-08T07:12:38.913 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-08T07:12:38.956 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-08T07:12:39.242 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pluggy. 2021-02-08T07:12:39.259 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-08T07:12:39.300 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-08T07:12:39.670 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-py. 2021-02-08T07:12:39.686 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-08T07:12:39.737 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-08T07:12:40.082 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pytest. 2021-02-08T07:12:40.097 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-08T07:12:40.140 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-08T07:12:40.494 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-attr. 2021-02-08T07:12:40.510 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-08T07:12:40.552 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-08T07:12:40.855 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cycler. 2021-02-08T07:12:40.872 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-08T07:12:40.914 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-08T07:12:41.242 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webencodings. 2021-02-08T07:12:41.257 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-08T07:12:41.300 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-08T07:12:41.620 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-html5lib. 2021-02-08T07:12:41.636 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-08T07:12:41.686 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-08T07:12:41.910 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-08T07:12:41.925 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:41.969 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libcephfs-dev (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:42.049 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-08T07:12:42.064 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-08T07:12:42.107 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-08T07:12:42.279 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package nvme-cli. 2021-02-08T07:12:42.294 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-08T07:12:42.346 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-08T07:12:42.552 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-08T07:12:42.565 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-08T07:12:42.610 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-08T07:12:42.676 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-attr. 2021-02-08T07:12:42.691 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-08T07:12:42.759 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-08T07:12:42.980 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tz. 2021-02-08T07:12:42.994 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-08T07:12:43.038 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-08T07:12:43.044 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-funcsigs. 2021-02-08T07:12:43.061 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-08T07:12:43.103 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-08T07:12:43.349 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-08T07:12:43.365 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-08T07:12:43.406 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-pluggy. 2021-02-08T07:12:43.407 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-08T07:12:43.420 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-08T07:12:43.464 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-08T07:12:43.817 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-py. 2021-02-08T07:12:43.832 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-08T07:12:43.875 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-08T07:12:44.097 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-olefile. 2021-02-08T07:12:44.109 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-08T07:12:44.146 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-08T07:12:44.229 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-pytest. 2021-02-08T07:12:44.244 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-08T07:12:44.287 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-08T07:12:44.474 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastescript. 2021-02-08T07:12:44.490 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-08T07:12:44.532 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-08T07:12:44.640 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-attr. 2021-02-08T07:12:44.654 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-08T07:12:44.698 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-08T07:12:44.878 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-08T07:12:44.895 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-08T07:12:44.944 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-08T07:12:45.010 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cycler. 2021-02-08T07:12:45.026 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-08T07:12:45.068 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-08T07:12:45.306 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pluggy. 2021-02-08T07:12:45.320 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-08T07:12:45.365 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-08T07:12:45.404 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-webencodings. 2021-02-08T07:12:45.419 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-08T07:12:45.462 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-08T07:12:45.701 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-py. 2021-02-08T07:12:45.717 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-08T07:12:45.767 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-08T07:12:45.773 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-html5lib. 2021-02-08T07:12:45.788 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-08T07:12:45.840 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-08T07:12:46.098 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-08T07:12:46.112 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-08T07:12:46.156 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-08T07:12:46.226 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-08T07:12:46.241 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-08T07:12:46.285 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-08T07:12:46.492 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pytest. 2021-02-08T07:12:46.508 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-08T07:12:46.550 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-08T07:12:46.730 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-08T07:12:46.746 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-08T07:12:46.787 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-08T07:12:46.896 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplejson. 2021-02-08T07:12:46.910 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-08T07:12:46.954 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-08T07:12:47.115 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-tz. 2021-02-08T07:12:47.132 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-08T07:12:47.173 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-08T07:12:47.282 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-08T07:12:47.297 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-08T07:12:47.340 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-08T07:12:47.535 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-08T07:12:47.549 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-08T07:12:47.601 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-08T07:12:47.860 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package radosgw. 2021-02-08T07:12:47.874 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../111-radosgw_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:47.927 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking radosgw (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:48.291 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-olefile. 2021-02-08T07:12:48.308 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-08T07:12:48.350 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-08T07:12:48.645 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pastescript. 2021-02-08T07:12:48.763 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-08T07:12:48.763 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-08T07:12:48.880 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package rbd-fuse. 2021-02-08T07:12:48.894 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../112-rbd-fuse_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:48.939 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking rbd-fuse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:49.097 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-08T07:12:49.112 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-08T07:12:49.164 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-08T07:12:49.257 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-08T07:12:49.551 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pluggy. 2021-02-08T07:12:49.569 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-08T07:12:49.593 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-08T07:12:49.609 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-08T07:12:49.903 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-py. 2021-02-08T07:12:49.921 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-08T07:12:49.962 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-08T07:12:49.969 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-08T07:12:50.095 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-08T07:12:50.282 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-08T07:12:50.297 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-08T07:12:50.365 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-08T07:12:50.676 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pytest. 2021-02-08T07:12:50.692 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-08T07:12:50.706 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-08T07:12:50.734 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-08T07:12:51.079 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-simplejson. 2021-02-08T07:12:51.091 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-08T07:12:51.094 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-08T07:12:51.137 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-08T07:12:51.375 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-08T07:12:51.490 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-08T07:12:51.505 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-08T07:12:51.548 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-08T07:12:51.660 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-08T07:12:51.870 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-08T07:12:52.002 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package radosgw. 2021-02-08T07:12:52.004 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-08T07:12:52.019 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../111-radosgw_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:52.052 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking radosgw (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:52.347 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-08T07:12:52.710 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-08T07:12:54.336 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package rbd-fuse. 2021-02-08T07:12:54.336 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../112-rbd-fuse_15.2.8-294-g133532e2-1bionic_amd64.deb ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking rbd-fuse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-argparse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-08T07:12:54.337 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-08T07:12:54.823 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-08T07:12:54.999 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-08T07:12:55.216 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-08T07:12:55.325 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-08T07:12:55.509 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-08T07:12:55.626 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-py (1.5.2-1) ... 2021-02-08T07:12:55.852 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-08T07:12:55.910 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-08T07:12:56.087 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-08T07:12:56.212 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-08T07:12:56.212 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-08T07:12:56.310 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-08T07:12:56.547 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-08T07:12:56.555 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-08T07:12:56.681 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-08T07:12:56.874 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-08T07:12:56.875 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-08T07:12:57.159 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-08T07:12:57.167 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-08T07:12:57.469 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-ceph-argparse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:57.493 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-08T07:12:57.761 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-08T07:12:57.786 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-08T07:12:58.046 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-08T07:12:58.170 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-08T07:12:58.171 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-08T07:12:58.513 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-08T07:12:58.815 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephadm (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:12:59.124 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-08T07:12:59.467 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-08T07:12:59.482 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user cephadm....done 2021-02-08T07:12:59.647 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-08T07:12:59.752 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-py (1.5.2-1) ... 2021-02-08T07:12:59.956 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-08T07:12:59.970 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-08T07:13:00.082 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-08T07:13:00.217 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-08T07:13:00.321 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-08T07:13:00.411 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-08T07:13:00.568 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-08T07:13:00.681 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-08T07:13:00.823 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-08T07:13:01.303 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-08T07:13:01.350 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-08T07:13:01.453 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-08T07:13:01.651 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-08T07:13:01.822 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-common (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:01.994 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-08T07:13:02.148 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-08T07:13:02.286 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-08T07:13:02.449 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-08T07:13:02.575 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-08T07:13:02.663 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-08T07:13:02.918 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-08T07:13:03.015 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-08T07:13:03.270 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-08T07:13:03.317 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up cephadm (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:03.376 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-08T07:13:03.408 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-08T07:13:03.647 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-08T07:13:03.948 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-08T07:13:03.958 INFO:teuthology.orchestra.run.smithi017.stdout:Adding system user cephadm....done 2021-02-08T07:13:04.123 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-08T07:13:04.282 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-08T07:13:04.457 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-08T07:13:04.583 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-08T07:13:04.617 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-08T07:13:04.718 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-08T07:13:04.934 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-08T07:13:05.078 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-08T07:13:05.135 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-08T07:13:05.378 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-08T07:13:05.530 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-08T07:13:05.562 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-08T07:13:05.652 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-08T07:13:05.873 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-ceph-common (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:05.906 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-08T07:13:06.083 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-08T07:13:06.207 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-08T07:13:06.209 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-08T07:13:06.469 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-08T07:13:06.492 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-08T07:13:06.603 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-08T07:13:06.618 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-08T07:13:06.969 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-08T07:13:07.047 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-08T07:13:07.173 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-08T07:13:07.321 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-08T07:13:07.410 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-08T07:13:07.443 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-08T07:13:07.475 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-08T07:13:07.681 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-08T07:13:07.776 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-08T07:13:07.982 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-08T07:13:08.077 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-08T07:13:08.308 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-08T07:13:08.403 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-08T07:13:08.664 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-08T07:13:08.796 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-08T07:13:08.986 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-08T07:13:09.107 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-08T07:13:09.178 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-08T07:13:09.416 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-08T07:13:09.580 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-08T07:13:09.672 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-08T07:13:09.727 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-08T07:13:09.927 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-08T07:13:10.103 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-08T07:13:10.104 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-08T07:13:10.221 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-08T07:13:10.438 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-08T07:13:10.480 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-08T07:13:10.564 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-08T07:13:10.623 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-08T07:13:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-08T07:13:11.168 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-08T07:13:11.469 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-08T07:13:11.728 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-08T07:13:11.871 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-08T07:13:12.071 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-08T07:13:12.172 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-08T07:13:12.406 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-08T07:13:12.482 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-08T07:13:12.532 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-08T07:13:12.859 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-08T07:13:12.909 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-08T07:13:13.143 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-08T07:13:13.169 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-08T07:13:13.244 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-08T07:13:13.345 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librados2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:13.470 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-08T07:13:13.496 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-08T07:13:13.788 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-08T07:13:13.814 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:13.940 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs-dev (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:14.066 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rados (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:14.157 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-08T07:13:14.167 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-08T07:13:14.460 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-08T07:13:14.466 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-08T07:13:14.592 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-08T07:13:14.752 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-08T07:13:15.062 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-08T07:13:15.789 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-08T07:13:16.008 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-08T07:13:16.099 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-08T07:13:16.443 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-08T07:13:16.577 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-08T07:13:16.804 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-08T07:13:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-08T07:13:17.130 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libradosstriper1 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:17.221 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-08T07:13:17.256 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cephfs (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:17.347 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-08T07:13:17.473 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librados2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:17.557 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librgw2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:17.625 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-08T07:13:17.675 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-fuse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:17.874 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-08T07:13:17.874 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-08T07:13:17.942 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libcephfs2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:18.093 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libcephfs-dev (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:18.126 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librbd1 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:18.219 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rados (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:18.227 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-08T07:13:18.301 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-08T07:13:18.341 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-08T07:13:18.345 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-08T07:13:18.656 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-08T07:13:18.663 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-08T07:13:18.965 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-08T07:13:19.290 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-08T07:13:19.311 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-modules-core (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:19.412 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-08T07:13:19.722 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rgw (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:19.848 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up rbd-fuse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:19.973 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rbd (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:20.116 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-common (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:20.295 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-08T07:13:20.558 INFO:teuthology.orchestra.run.smithi110.stdout:Adding group ceph....done 2021-02-08T07:13:20.978 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user ceph....done 2021-02-08T07:13:21.149 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-08T07:13:21.279 INFO:teuthology.orchestra.run.smithi110.stdout:Setting system user ceph properties....done 2021-02-08T07:13:21.444 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-08T07:13:21.450 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libradosstriper1 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:21.576 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cephfs (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:21.675 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-08T07:13:21.895 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librgw2 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:21.943 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up radosgw (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:22.021 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-fuse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:22.217 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-08T07:13:22.217 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-08T07:13:22.238 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-08T07:13:22.238 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-08T07:13:22.531 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librbd1 (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:22.660 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-08T07:13:22.671 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-test (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:22.749 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-08T07:13:22.771 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-base (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:22.790 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-08T07:13:23.089 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-08T07:13:23.130 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-08T07:13:23.381 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:23.602 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-08T07:13:23.602 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-08T07:13:23.790 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-modules-core (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:23.849 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-diskprediction-local (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:23.916 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-08T07:13:23.992 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-osd (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:24.242 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rgw (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:24.373 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up rbd-fuse (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:24.394 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-08T07:13:24.394 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-08T07:13:24.528 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rbd (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:24.652 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-cephadm (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:24.662 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-common (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:24.770 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:24.887 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-rook (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:25.005 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mds (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:25.164 INFO:teuthology.orchestra.run.smithi017.stdout:Adding group ceph....done 2021-02-08T07:13:25.213 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-08T07:13:25.213 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-08T07:13:25.481 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mon (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:25.566 INFO:teuthology.orchestra.run.smithi017.stdout:Adding system user ceph....done 2021-02-08T07:13:25.696 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-08T07:13:25.696 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-08T07:13:25.784 INFO:teuthology.orchestra.run.smithi017.stdout:Setting system user ceph properties....done 2021-02-08T07:13:25.946 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-08T07:13:25.966 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-k8sevents (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:26.076 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-dashboard (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:26.185 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-08T07:13:26.194 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:26.405 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-08T07:13:26.464 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up radosgw (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:26.564 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-08T07:13:26.830 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-08T07:13:26.830 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-08T07:13:27.267 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-test (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:27.393 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-base (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:27.749 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-08T07:13:28.053 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:28.279 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-08T07:13:28.279 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-08T07:13:28.521 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-diskprediction-local (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:28.608 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-08T07:13:28.652 INFO:teuthology.orchestra.run.smithi110.stdout:ureadahead will be reprofiled on next reboot 2021-02-08T07:13:28.655 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-osd (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:28.709 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-08T07:13:29.104 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-08T07:13:29.104 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-08T07:13:29.170 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-08T07:13:29.172 DEBUG:teuthology.parallel:result is None 2021-02-08T07:13:29.399 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-cephadm (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:29.559 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:29.711 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-rook (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:29.853 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mds (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:30.084 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-08T07:13:30.084 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-08T07:13:30.380 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mon (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:30.601 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-08T07:13:30.601 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-08T07:13:30.898 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-k8sevents (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:31.024 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-dashboard (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:31.166 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph (15.2.8-294-g133532e2-1bionic) ... 2021-02-08T07:13:31.293 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-08T07:13:31.502 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-08T07:13:33.630 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-08T07:13:33.673 INFO:teuthology.orchestra.run.smithi017.stdout:ureadahead will be reprofiled on next reboot 2021-02-08T07:13:33.723 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-08T07:13:34.150 INFO:teuthology.orchestra.run.smithi017.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-08T07:13:34.152 DEBUG:teuthology.parallel:result is None 2021-02-08T07:13:34.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo lsb_release -is 2021-02-08T07:13:34.197 INFO:teuthology.orchestra.run.smithi017.stdout:Ubuntu 2021-02-08T07:13:34.201 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-08T07:13:34.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=133532e25eccb46507b425783df68c050462a609 2021-02-08T07:13:34.384 DEBUG:teuthology.orchestra.run.smithi017:> dpkg-query -W -f '${Version}' ceph 2021-02-08T07:13:34.409 INFO:teuthology.orchestra.run.smithi017.stdout:15.2.8-294-g133532e2-1bionic 2021-02-08T07:13:34.410 INFO:teuthology.packaging:The installed version of ceph is 15.2.8-294-g133532e2-1bionic 2021-02-08T07:13:34.410 INFO:teuthology.task.install:The correct ceph version 15.2.8-294-g133532e2-1bionic is installed. 2021-02-08T07:13:34.410 DEBUG:teuthology.orchestra.run.smithi110:> sudo lsb_release -is 2021-02-08T07:13:34.468 INFO:teuthology.orchestra.run.smithi110.stdout:Ubuntu 2021-02-08T07:13:34.473 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-08T07:13:34.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=133532e25eccb46507b425783df68c050462a609 2021-02-08T07:13:34.631 DEBUG:teuthology.orchestra.run.smithi110:> dpkg-query -W -f '${Version}' ceph 2021-02-08T07:13:34.656 INFO:teuthology.orchestra.run.smithi110.stdout:15.2.8-294-g133532e2-1bionic 2021-02-08T07:13:34.656 INFO:teuthology.packaging:The installed version of ceph is 15.2.8-294-g133532e2-1bionic 2021-02-08T07:13:34.657 INFO:teuthology.task.install:The correct ceph version 15.2.8-294-g133532e2-1bionic is installed. 2021-02-08T07:13:34.657 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-08T07:13:34.657 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:13:34.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-08T07:13:34.673 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:13:34.673 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-08T07:13:34.718 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-08T07:13:34.718 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:13:34.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/daemon-helper 2021-02-08T07:13:34.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-08T07:13:34.789 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:13:34.789 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2021-02-08T07:13:34.803 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-08T07:13:34.861 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-08T07:13:34.861 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:13:34.861 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-08T07:13:34.877 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-08T07:13:34.936 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:13:34.936 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-08T07:13:34.949 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-08T07:13:35.005 INFO:teuthology.run_tasks:Running task ceph... 2021-02-08T07:13:35.072 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-02-08T07:13:35.072 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /var/log/ceph 2021-02-08T07:13:35.074 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2021-02-08T07:13:35.089 INFO:tasks.ceph:Disabling ceph logrotate... 2021-02-08T07:13:35.090 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-08T07:13:35.132 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-08T07:13:35.145 INFO:tasks.ceph:Creating extra log directories... 2021-02-08T07:13:35.146 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-08T07:13:35.188 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-08T07:13:35.201 INFO:tasks.ceph:Creating ceph cluster c1... 2021-02-08T07:13:35.202 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-08T07:13:35.202 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864577', 'branch': 'wip-yuri3-testing-2021-02-03-1046-octopus', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5864577', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2021-02-03-1046-octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '133532e25eccb46507b425783df68c050462a609', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '133532e25eccb46507b425783df68c050462a609'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuri3-testing-2021-02-03-1046-octopus', 'sha1': '133532e25eccb46507b425783df68c050462a609'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '133532e25eccb46507b425783df68c050462a609', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri3-testing-2021-02-03-1046-octopus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_133532e25eccb46507b425783df68c050462a609/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '133532e25eccb46507b425783df68c050462a609', 'targets': {'smithi017.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqsworecZ+NXvz4RgtnimxK4DRM14Ko8sQhTWWw6hOfRi9MqTDMncDlRIIOghbLxvFInRuOuHi/3+gpd77/1ytqJx7zC+eCamEq0Mc3j1fBxDInmIyIuctj/+rpolBZnsyXtPREEPTLrXPMufVqWHV8bxtYuVelT+PKbvTLdKuawY4l1oxsDSr5aJ4aHBA1FnAe15iFNNbQzsbbi0tObW1ag3znNRh5+XUIFwuJZJbhMe4DUFSdE2/Xc+2EQ6MmlvvylPN9vjVJ1hu0aihPEPK0HJbpbk/u3aC/hlKu48Z7mokBux4Nmt9U/Jr0p02cENBoxmE//NCAZApA8SHYCil', 'smithi110.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDT7W0N7K3cyUIC6+xOIqn6tGKNOr8qH8UlFgikfk6WJjcL5Dpz/7AZUsa109iARI6teTVv/APcQ0Q/Cy/eXRmD9EfjCc/7i52q0S0muTTdQtLeCwn6F7C47x0KZobHgG/LWZY99i/OsL29ycoraHNTeR8DOM2hS47MxIbkQVykN4+EjG1GNIltOL5s8yfy6M2L+WMALpBRJhL5H+7cU9B7Sn+SG5IJ/bqzbtsonVr2CCe0QRBHAEtbxw+kNVw8MEdP01VR9QB05RV059xowBtLhUUWFJJ5F9sy5T072VYD+x5AKKdt51TagnD0mjq3kffa6AOURoQo4fBI7l9Q2nRz'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '133532e25eccb46507b425783df68c050462a609', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '17c099650191e7aa47a7d9f502facca74f9b4d44', 'timestamp': '2021-02-07_16:16:23', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-08T07:13:35.202 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-08T07:13:35.243 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-08T07:13:35.251 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2021-02-08T07:13:35.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2021-02-08T07:13:35.309 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:13:35.310 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2021-02-08T07:13:35.352 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-08T07:13:35.352 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-08T07:13:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-08T07:13:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-08T07:13:35.402 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-02-08T07:13:35.402 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-08T07:13:35.402 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2021-02-08 07:10:58.903273353 +0000 2021-02-08T07:13:35.402 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2021-02-08 07:10:58.759201356 +0000 2021-02-08T07:13:35.402 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2021-02-08 07:10:58.759201356 +0000 2021-02-08T07:13:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2021-02-08T07:13:35.411 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2021-02-08T07:13:35.411 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2021-02-08T07:13:35.412 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000388965 s, 1.3 MB/s 2021-02-08T07:13:35.416 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-08T07:13:35.465 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-08T07:13:35.465 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-08T07:13:35.465 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 347 Links: 1 2021-02-08T07:13:35.465 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-08T07:13:35.465 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2021-02-08 07:10:59.167405348 +0000 2021-02-08T07:13:35.465 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2021-02-08 07:10:59.039341351 +0000 2021-02-08T07:13:35.465 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2021-02-08 07:10:59.039341351 +0000 2021-02-08T07:13:35.465 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2021-02-08T07:13:35.475 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2021-02-08T07:13:35.475 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2021-02-08T07:13:35.475 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000354355 s, 1.4 MB/s 2021-02-08T07:13:35.479 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-08T07:13:35.528 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-08T07:13:35.528 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-08T07:13:35.529 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 353 Links: 1 2021-02-08T07:13:35.529 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-08T07:13:35.529 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2021-02-08 07:10:59.427535342 +0000 2021-02-08T07:13:35.529 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2021-02-08 07:10:59.295469345 +0000 2021-02-08T07:13:35.529 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2021-02-08 07:10:59.295469345 +0000 2021-02-08T07:13:35.529 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2021-02-08T07:13:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2021-02-08T07:13:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2021-02-08T07:13:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.00038352 s, 1.3 MB/s 2021-02-08T07:13:35.541 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-08T07:13:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-08T07:13:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-08T07:13:35.589 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 359 Links: 1 2021-02-08T07:13:35.589 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-08T07:13:35.589 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2021-02-08 07:10:59.675659337 +0000 2021-02-08T07:13:35.589 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2021-02-08 07:10:59.559601339 +0000 2021-02-08T07:13:35.589 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2021-02-08 07:10:59.559601339 +0000 2021-02-08T07:13:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2021-02-08T07:13:35.597 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2021-02-08T07:13:35.597 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2021-02-08T07:13:35.598 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000403955 s, 1.3 MB/s 2021-02-08T07:13:35.601 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-08T07:13:35.601 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-08T07:13:35.601 INFO:tasks.ceph:Generating config... 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-08T07:13:35.603 INFO:tasks.ceph:[client] setuser = ceph 2021-02-08T07:13:35.604 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-08T07:13:35.604 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-08T07:13:35.604 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-08T07:13:35.604 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-08T07:13:35.604 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-08T07:13:35.604 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-08T07:13:35.604 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-08T07:13:35.604 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-08T07:13:35.604 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-08T07:13:35.604 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-08T07:13:35.604 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-08T07:13:35.604 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-08T07:13:35.604 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-08T07:13:35.604 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-08T07:13:35.605 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-08T07:13:35.605 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-08T07:13:35.605 INFO:tasks.ceph:Setting up c1.mon.a... 2021-02-08T07:13:35.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-02-08T07:13:35.673 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/c1.keyring 2021-02-08T07:13:35.675 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-02-08T07:13:35.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-08T07:13:35.808 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.17')] 2021-02-08T07:13:35.809 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.17 --print /home/ubuntu/cephtest/c1.monmap 2021-02-08T07:13:35.878 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-02-08T07:13:35.878 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: generated fsid cc72d099-b5fb-4f3a-9eb6-7ad41378824e 2021-02-08T07:13:35.878 INFO:teuthology.orchestra.run.smithi017.stdout:epoch 0 2021-02-08T07:13:35.878 INFO:teuthology.orchestra.run.smithi017.stdout:fsid cc72d099-b5fb-4f3a-9eb6-7ad41378824e 2021-02-08T07:13:35.878 INFO:teuthology.orchestra.run.smithi017.stdout:last_changed 2021-02-08T07:13:35.879215+0000 2021-02-08T07:13:35.879 INFO:teuthology.orchestra.run.smithi017.stdout:created 2021-02-08T07:13:35.879215+0000 2021-02-08T07:13:35.879 INFO:teuthology.orchestra.run.smithi017.stdout:min_mon_release 0 (unknown) 2021-02-08T07:13:35.879 INFO:teuthology.orchestra.run.smithi017.stdout:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2021-02-08T07:13:35.879 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-02-08T07:13:35.880 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID cc72d099-b5fb-4f3a-9eb6-7ad41378824e... 2021-02-08T07:13:35.881 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-08T07:13:35.924 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-08T07:13:35.953 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2021-02-08T07:13:35.953 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2021-02-08T07:13:35.953 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-08T07:13:35.953 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2021-02-08T07:13:35.953 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.954 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-08T07:13:35.955 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = 172.21.15.17 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: osd heartbeat grace = 40 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = cc72d099-b5fb-4f3a-9eb6-7ad41378824e 2021-02-08T07:13:35.956 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2021-02-08T07:13:35.957 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.958 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: debug filestore = 20 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: debug journal = 20 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 20 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 25 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: osd fast shutdown = False 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: rocksdb delete range threshold = 0 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2021-02-08T07:13:35.959 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2021-02-08T07:13:35.960 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd crush smoke test = False 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: debug rgw = 20 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: rgw crypt require ssl = False 2021-02-08T07:13:35.961 INFO:teuthology.orchestra.run.smithi110.stdout: rgw crypt s3 kms backend = testing 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: rgw curl low speed time = 300 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: rgw data log num shards = 4 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: rgw md log max shards = 4 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: rgw sync log trim interval = 0 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: rgw sync obj etag verify = True 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: setgroup = ceph 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: setuser = ceph 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: osd_max_omap_entries_per_request = 10 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout: osd_max_omap_entries_per_request = 1000 2021-02-08T07:13:35.962 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2021-02-08T07:13:35.963 INFO:teuthology.orchestra.run.smithi110.stdout: osd_max_omap_entries_per_request = 10000 2021-02-08T07:13:35.963 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2021-02-08T07:13:35.964 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2021-02-08T07:13:35.965 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2021-02-08T07:13:35.966 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: mon health detail to clog = false 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = 172.21.15.17 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: osd heartbeat grace = 40 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = cc72d099-b5fb-4f3a-9eb6-7ad41378824e 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2021-02-08T07:13:35.967 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.968 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: debug filestore = 20 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: debug journal = 20 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 20 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 25 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: osd fast shutdown = False 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: rocksdb delete range threshold = 0 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2021-02-08T07:13:35.969 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2021-02-08T07:13:35.970 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd crush smoke test = False 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: debug rgw = 20 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt require ssl = False 2021-02-08T07:13:35.971 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms backend = testing 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: rgw curl low speed time = 300 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: rgw data log num shards = 4 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: rgw md log max shards = 4 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync log trim interval = 0 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync obj etag verify = True 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: setgroup = ceph 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: setuser = ceph 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 1000 2021-02-08T07:13:35.972 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2021-02-08T07:13:35.973 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10000 2021-02-08T07:13:35.973 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-02-08T07:13:35.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-02-08T07:13:36.059 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-08T07:13:36.066 DEBUG:teuthology.orchestra.remote:smithi017:/etc/ceph/c1.keyring is 206B 2021-02-08T07:13:36.124 DEBUG:teuthology.orchestra.remote:smithi017:/home/ubuntu/cephtest/c1.monmap is 179B 2021-02-08T07:13:36.135 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2021-02-08T07:13:36.135 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:13:36.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/c1.keyring 2021-02-08T07:13:36.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-08T07:13:36.160 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:13:36.160 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-08T07:13:36.210 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2021-02-08T07:13:36.211 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:13:36.211 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/c1.keyring 2021-02-08T07:13:36.211 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-08T07:13:36.233 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:13:36.233 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-08T07:13:36.281 INFO:tasks.ceph:Setting up mon nodes... 2021-02-08T07:13:36.281 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-08T07:13:36.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-02-08T07:13:36.322 INFO:teuthology.orchestra.run.smithi017.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-02-08T07:13:36.324 INFO:tasks.ceph:Setting up mds nodes... 2021-02-08T07:13:36.325 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-08T07:13:36.325 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-02-08T07:13:36.400 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/c1.client.0.keyring 2021-02-08T07:13:36.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-02-08T07:13:36.489 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/c1.client.1.keyring 2021-02-08T07:13:36.502 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-08T07:13:36.502 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-08T07:13:36.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-02-08T07:13:36.561 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-08T07:13:36.561 INFO:tasks.ceph:role: c1.osd.0 2021-02-08T07:13:36.561 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com 2021-02-08T07:13:36.561 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-08T07:13:37.063 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-08T07:13:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-08T07:13:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-08T07:13:37.064 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-08T07:13:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2021-02-08T07:13:37.064 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-08T07:13:37.064 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-08T07:13:37.064 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-08T07:13:37.064 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-08T07:13:37.065 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2021-02-08T07:13:37.065 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-02-08T07:13:37.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-02-08T07:13:37.163 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: /sbin/restorecon: command not found 2021-02-08T07:13:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-08T07:13:37.164 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-02-08T07:13:37.221 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-08T07:13:37.221 INFO:tasks.ceph:role: c1.osd.1 2021-02-08T07:13:37.221 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com 2021-02-08T07:13:37.222 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-08T07:13:37.728 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-08T07:13:37.728 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-08T07:13:37.728 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-08T07:13:37.728 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-08T07:13:37.728 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2021-02-08T07:13:37.728 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-08T07:13:37.728 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-08T07:13:37.728 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-08T07:13:37.729 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-08T07:13:37.729 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2021-02-08T07:13:37.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-02-08T07:13:37.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-02-08T07:13:37.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: /sbin/restorecon: command not found 2021-02-08T07:13:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-08T07:13:37.816 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-02-08T07:13:37.873 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-08T07:13:37.873 INFO:tasks.ceph:role: c1.osd.2 2021-02-08T07:13:37.873 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com 2021-02-08T07:13:37.873 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-08T07:13:38.377 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-08T07:13:38.377 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-08T07:13:38.378 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-08T07:13:38.378 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-08T07:13:38.378 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2021-02-08T07:13:38.378 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-08T07:13:38.378 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-08T07:13:38.378 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-08T07:13:38.378 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-08T07:13:38.379 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2021-02-08T07:13:38.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-02-08T07:13:38.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-02-08T07:13:38.439 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: /sbin/restorecon: command not found 2021-02-08T07:13:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-08T07:13:38.439 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-08T07:13:38.536 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:38.535+0000 7f4d6bd21d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-02-08T07:13:38.537 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:38.535+0000 7f4d6bd21d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-02-08T07:13:38.537 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:38.535+0000 7f4d6bd21d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-08T07:13:38.537 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:38.535+0000 7f4d6bd21d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-08T07:13:38.537 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:38.535+0000 7f4d6bd21d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-08T07:13:38.539 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:38.539+0000 7f4d6bd21d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-02-08T07:13:38.557 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:38.555+0000 7f4d6bd21d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-02-08T07:13:40.449 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-02-08T07:13:40.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-08T07:13:40.544 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:40.543+0000 7fa729542d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-02-08T07:13:40.544 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:40.543+0000 7fa729542d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-02-08T07:13:40.544 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:40.543+0000 7fa729542d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-08T07:13:40.544 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:40.543+0000 7fa729542d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-08T07:13:40.545 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:40.543+0000 7fa729542d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-08T07:13:40.545 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:40.543+0000 7fa729542d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-02-08T07:13:40.560 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:40.559+0000 7fa729542d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-02-08T07:13:42.465 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-02-08T07:13:42.521 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-08T07:13:42.600 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:42.599+0000 7f5f11fcfd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-02-08T07:13:42.600 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:42.599+0000 7f5f11fcfd80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-02-08T07:13:42.600 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:42.599+0000 7f5f11fcfd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-08T07:13:42.600 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:42.599+0000 7f5f11fcfd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-08T07:13:42.600 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:42.599+0000 7f5f11fcfd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-08T07:13:42.601 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:42.599+0000 7f5f11fcfd80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-02-08T07:13:42.614 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:13:42.611+0000 7f5f11fcfd80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-02-08T07:13:44.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-02-08T07:13:44.577 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-08T07:13:44.578 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2021-02-08T07:13:44.624 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.UWwDxFtFN1 2021-02-08T07:13:44.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.UWwDxFtFN1 2021-02-08T07:13:44.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.UWwDxFtFN1 2021-02-08T07:13:44.786 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.UWwDxFtFN1 is 56B 2021-02-08T07:13:44.797 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.UWwDxFtFN1 2021-02-08T07:13:44.803 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2021-02-08T07:13:44.851 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.lzOrEAC8Tk 2021-02-08T07:13:44.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.lzOrEAC8Tk 2021-02-08T07:13:44.909 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.lzOrEAC8Tk 2021-02-08T07:13:45.014 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.lzOrEAC8Tk is 56B 2021-02-08T07:13:45.025 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.lzOrEAC8Tk 2021-02-08T07:13:45.031 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2021-02-08T07:13:45.079 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.rOYq5LFFod 2021-02-08T07:13:45.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.rOYq5LFFod 2021-02-08T07:13:45.137 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.rOYq5LFFod 2021-02-08T07:13:45.242 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.rOYq5LFFod is 56B 2021-02-08T07:13:45.252 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.rOYq5LFFod 2021-02-08T07:13:45.258 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2021-02-08T07:13:45.307 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.lTPasEdeXD 2021-02-08T07:13:45.308 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.lTPasEdeXD 2021-02-08T07:13:45.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.lTPasEdeXD 2021-02-08T07:13:45.469 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.lTPasEdeXD is 56B 2021-02-08T07:13:45.481 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.lTPasEdeXD 2021-02-08T07:13:45.538 DEBUG:teuthology.orchestra.remote:smithi017:/etc/ceph/c1.client.0.keyring is 59B 2021-02-08T07:13:45.556 DEBUG:teuthology.orchestra.remote:smithi017:/etc/ceph/c1.client.1.keyring is 59B 2021-02-08T07:13:45.566 INFO:tasks.ceph:Adding keys to all mons... 2021-02-08T07:13:45.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo tee -a /etc/ceph/c1.keyring 2021-02-08T07:13:45.610 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr.x] 2021-02-08T07:13:45.610 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCg5CBg4YNGExAAc8qsqfrnhjG9YAa1kc1TZA== 2021-02-08T07:13:45.610 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2021-02-08T07:13:45.610 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCi5CBgMagDIBAAPaSgA3WcB8v/cNdbNlUFfA== 2021-02-08T07:13:45.611 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2021-02-08T07:13:45.611 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCk5CBglH96IBAAVkyA5bSetBhcehlw2x/ISw== 2021-02-08T07:13:45.611 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2021-02-08T07:13:45.611 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCm5CBgwtzIIxAARVe5Ac2XdqXETMQ9qmdNcQ== 2021-02-08T07:13:45.611 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2021-02-08T07:13:45.611 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCg5CBgNcnqFxAAV0NSIyQGR67/L3PuVq//wQ== 2021-02-08T07:13:45.611 INFO:teuthology.orchestra.run.smithi017.stdout:[client.1] 2021-02-08T07:13:45.611 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCg5CBgClE3HRAAYJTDTXGXuGgq1Co5VjYYLg== 2021-02-08T07:13:45.612 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-08T07:13:45.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-08T07:13:45.788 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-08T07:13:45.872 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-08T07:13:45.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-08T07:13:46.043 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-08T07:13:46.082 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-08T07:13:46.083 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-02-08T07:13:46.140 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-02-08T07:13:46.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-02-08T07:13:46.251 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-02-08T07:13:46.299 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-02-08T07:13:46.300 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-02-08T07:13:46.300 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-02-08T07:13:46.343 INFO:tasks.ceph.c1.mon.a:Started 2021-02-08T07:13:46.344 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-02-08T07:13:46.344 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-02-08T07:13:46.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-02-08T07:13:46.346 INFO:tasks.ceph.c1.mgr.x:Started 2021-02-08T07:13:46.346 INFO:tasks.ceph:Setting crush tunables to default 2021-02-08T07:13:46.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd crush tunables default 2021-02-08T07:13:46.923 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2021-02-08T07:13:46.933 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-02-08T07:13:46.934 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2021-02-08T07:13:46.940 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.zqtEZZ8ZWi 2021-02-08T07:13:46.940 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.zqtEZZ8ZWi 2021-02-08T07:13:46.996 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.zqtEZZ8ZWi 2021-02-08T07:13:47.101 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.zqtEZZ8ZWi is 37B 2021-02-08T07:13:47.154 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.zqtEZZ8ZWi 2021-02-08T07:13:47.159 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2021-02-08T07:13:47.211 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.oef43H2ma5 2021-02-08T07:13:47.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.oef43H2ma5 2021-02-08T07:13:47.268 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.oef43H2ma5 2021-02-08T07:13:47.372 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.oef43H2ma5 is 37B 2021-02-08T07:13:47.422 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.oef43H2ma5 2021-02-08T07:13:47.427 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2021-02-08T07:13:47.475 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.DqcxK9eO3i 2021-02-08T07:13:47.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.DqcxK9eO3i 2021-02-08T07:13:47.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.DqcxK9eO3i 2021-02-08T07:13:47.638 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.DqcxK9eO3i is 37B 2021-02-08T07:13:47.648 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.DqcxK9eO3i 2021-02-08T07:13:47.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd new 8765b53a-d24c-469b-9941-e8010b54fac0 0 2021-02-08T07:13:47.915 INFO:teuthology.orchestra.run.smithi017.stdout:0 2021-02-08T07:13:47.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd new 295ae8ad-356e-453c-8b91-63f53a1ee639 1 2021-02-08T07:13:48.120 INFO:teuthology.orchestra.run.smithi017.stdout:1 2021-02-08T07:13:48.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd new e50c2a1e-7f9e-4803-b418-e50f3fc4c612 2 2021-02-08T07:13:48.331 INFO:teuthology.orchestra.run.smithi017.stdout:2 2021-02-08T07:13:48.350 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-02-08T07:13:48.350 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-02-08T07:13:48.352 INFO:tasks.ceph.c1.osd.0:Started 2021-02-08T07:13:48.352 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-02-08T07:13:48.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-02-08T07:13:48.354 INFO:tasks.ceph.c1.osd.1:Started 2021-02-08T07:13:48.354 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-02-08T07:13:48.355 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-02-08T07:13:48.356 INFO:tasks.ceph.c1.osd.2:Started 2021-02-08T07:13:48.356 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-08T07:13:48.659 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-02-08T07:13:48.835 INFO:teuthology.misc.health.smithi017.stdout: 2021-02-08T07:13:48.835 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":5,"fsid":"cc72d099-b5fb-4f3a-9eb6-7ad41378824e","created":"2021-02-08T07:13:46.473218+0000","modified":"2021-02-08T07:13:48.329362+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"8765b53a-d24c-469b-9941-e8010b54fac0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"295ae8ad-356e-453c-8b91-63f53a1ee639","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e50c2a1e-7f9e-4803-b418-e50f3fc4c612","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:13:48.846 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-08T07:13:48.952 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2021-02-08T07:13:48.950+0000 7fde946d4d80 -1 Falling back to public interface 2021-02-08T07:13:48.963 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2021-02-08T07:13:48.962+0000 7f0b683d4d80 -1 Falling back to public interface 2021-02-08T07:13:48.964 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2021-02-08T07:13:48.962+0000 7fc61af4cd80 -1 Falling back to public interface 2021-02-08T07:13:49.574 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2021-02-08T07:13:49.570+0000 7fde946d4d80 -1 osd.2 0 log_to_monitors {default=true} 2021-02-08T07:13:49.580 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2021-02-08T07:13:49.578+0000 7f0b683d4d80 -1 osd.0 0 log_to_monitors {default=true} 2021-02-08T07:13:49.601 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2021-02-08T07:13:49.598+0000 7fc61af4cd80 -1 osd.1 0 log_to_monitors {default=true} 2021-02-08T07:13:51.517 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2021-02-08T07:13:51.514+0000 7f0b51e20700 -1 osd.0 0 waiting for initial osdmap 2021-02-08T07:13:51.517 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2021-02-08T07:13:51.514+0000 7fc604998700 -1 osd.1 0 waiting for initial osdmap 2021-02-08T07:13:51.517 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2021-02-08T07:13:51.514+0000 7fde7e120700 -1 osd.2 0 waiting for initial osdmap 2021-02-08T07:13:55.150 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-02-08T07:13:55.421 INFO:teuthology.misc.health.smithi017.stdout: 2021-02-08T07:13:55.421 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"cc72d099-b5fb-4f3a-9eb6-7ad41378824e","created":"2021-02-08T07:13:46.473218+0000","modified":"2021-02-08T07:13:54.528958+0000","last_up_change":"2021-02-08T07:13:52.511929+0000","last_in_change":"2021-02-08T07:13:52.511929+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-08T07:13:50.824449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8765b53a-d24c-469b-9941-e8010b54fac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":12460},{"type":"v1","addr":"172.21.15.17:6809","nonce":12460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":12460},{"type":"v1","addr":"172.21.15.17:6812","nonce":12460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6817","nonce":12460},{"type":"v1","addr":"172.21.15.17:6818","nonce":12460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6813","nonce":12460},{"type":"v1","addr":"172.21.15.17:6815","nonce":12460}]},"public_addr":"172.21.15.17:6809/12460","cluster_addr":"172.21.15.17:6812/12460","heartbeat_back_addr":"172.21.15.17:6818/12460","heartbeat_front_addr":"172.21.15.17:6815/12460","state":["exists","up"]},{"osd":1,"uuid":"295ae8ad-356e-453c-8b91-63f53a1ee639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":12462},{"type":"v1","addr":"172.21.15.17:6814","nonce":12462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":12462},{"type":"v1","addr":"172.21.15.17:6819","nonce":12462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":12462},{"type":"v1","addr":"172.21.15.17:6823","nonce":12462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":12462},{"type":"v1","addr":"172.21.15.17:6821","nonce":12462}]},"public_addr":"172.21.15.17:6814/12462","cluster_addr":"172.21.15.17:6819/12462","heartbeat_back_addr":"172.21.15.17:6823/12462","heartbeat_front_addr":"172.21.15.17:6821/12462","state":["exists","up"]},{"osd":2,"uuid":"e50c2a1e-7f9e-4803-b418-e50f3fc4c612","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":12461},{"type":"v1","addr":"172.21.15.17:6801","nonce":12461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":12461},{"type":"v1","addr":"172.21.15.17:6803","nonce":12461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":12461},{"type":"v1","addr":"172.21.15.17:6807","nonce":12461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":12461},{"type":"v1","addr":"172.21.15.17:6805","nonce":12461}]},"public_addr":"172.21.15.17:6801/12461","cluster_addr":"172.21.15.17:6803/12461","heartbeat_back_addr":"172.21.15.17:6807/12461","heartbeat_front_addr":"172.21.15.17:6805/12461","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.610648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.593456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.540948+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:13:55.434 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-08T07:13:55.434 INFO:tasks.ceph:Creating RBD pool 2021-02-08T07:13:55.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd pool create rbd 8 2021-02-08T07:13:56.550 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'rbd' created 2021-02-08T07:13:56.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-08T07:13:57.559 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-08T07:13:57.574 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-02-08T07:13:57.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2021-02-08T07:13:57.575 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-08T07:13:57.842 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:57.842 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":12,"fsid":"cc72d099-b5fb-4f3a-9eb6-7ad41378824e","created":"2021-02-08T07:13:46.473218+0000","modified":"2021-02-08T07:13:57.544170+0000","last_up_change":"2021-02-08T07:13:52.511929+0000","last_in_change":"2021-02-08T07:13:52.511929+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-08T07:13:50.824449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-08T07:13:55.700003+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8765b53a-d24c-469b-9941-e8010b54fac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":12460},{"type":"v1","addr":"172.21.15.17:6809","nonce":12460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":12460},{"type":"v1","addr":"172.21.15.17:6812","nonce":12460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6817","nonce":12460},{"type":"v1","addr":"172.21.15.17:6818","nonce":12460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6813","nonce":12460},{"type":"v1","addr":"172.21.15.17:6815","nonce":12460}]},"public_addr":"172.21.15.17:6809/12460","cluster_addr":"172.21.15.17:6812/12460","heartbeat_back_addr":"172.21.15.17:6818/12460","heartbeat_front_addr":"172.21.15.17:6815/12460","state":["exists","up"]},{"osd":1,"uuid":"295ae8ad-356e-453c-8b91-63f53a1ee639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":12462},{"type":"v1","addr":"172.21.15.17:6814","nonce":12462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":12462},{"type":"v1","addr":"172.21.15.17:6819","nonce":12462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":12462},{"type":"v1","addr":"172.21.15.17:6823","nonce":12462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":12462},{"type":"v1","addr":"172.21.15.17:6821","nonce":12462}]},"public_addr":"172.21.15.17:6814/12462","cluster_addr":"172.21.15.17:6819/12462","heartbeat_back_addr":"172.21.15.17:6823/12462","heartbeat_front_addr":"172.21.15.17:6821/12462","state":["exists","up"]},{"osd":2,"uuid":"e50c2a1e-7f9e-4803-b418-e50f3fc4c612","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":12461},{"type":"v1","addr":"172.21.15.17:6801","nonce":12461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":12461},{"type":"v1","addr":"172.21.15.17:6803","nonce":12461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":12461},{"type":"v1","addr":"172.21.15.17:6807","nonce":12461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":12461},{"type":"v1","addr":"172.21.15.17:6805","nonce":12461}]},"public_addr":"172.21.15.17:6801/12461","cluster_addr":"172.21.15.17:6803/12461","heartbeat_back_addr":"172.21.15.17:6807/12461","heartbeat_front_addr":"172.21.15.17:6805/12461","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.610648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.593456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.540948+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:13:57.856 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-08T07:13:50.824449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-08T07:13:55.700003+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-08T07:13:57.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2021-02-08T07:13:58.125 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2021-02-08T07:13:58.138 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2021-02-08T07:13:58.414 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 8 2021-02-08T07:13:58.427 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-02-08T07:13:58.427 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-02-08T07:13:58.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2021-02-08T07:13:58.729 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:58.746 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":12251},{"type":"v1","addr":"172.21.15.17:6825","nonce":12251}]},"active_addr":"172.21.15.17:6825/12251","active_change":"2021-02-08T07:13:50.793082+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":1711533838}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":4197400048}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":1503711849}]}]}} 2021-02-08T07:13:58.748 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-02-08T07:13:58.748 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-02-08T07:13:58.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2021-02-08T07:13:59.012 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:59.012 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":13,"fsid":"cc72d099-b5fb-4f3a-9eb6-7ad41378824e","created":"2021-02-08T07:13:46.473218+0000","modified":"2021-02-08T07:13:58.553964+0000","last_up_change":"2021-02-08T07:13:52.511929+0000","last_in_change":"2021-02-08T07:13:52.511929+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-08T07:13:50.824449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-08T07:13:55.700003+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8765b53a-d24c-469b-9941-e8010b54fac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":12460},{"type":"v1","addr":"172.21.15.17:6809","nonce":12460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":12460},{"type":"v1","addr":"172.21.15.17:6812","nonce":12460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6817","nonce":12460},{"type":"v1","addr":"172.21.15.17:6818","nonce":12460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6813","nonce":12460},{"type":"v1","addr":"172.21.15.17:6815","nonce":12460}]},"public_addr":"172.21.15.17:6809/12460","cluster_addr":"172.21.15.17:6812/12460","heartbeat_back_addr":"172.21.15.17:6818/12460","heartbeat_front_addr":"172.21.15.17:6815/12460","state":["exists","up"]},{"osd":1,"uuid":"295ae8ad-356e-453c-8b91-63f53a1ee639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":12462},{"type":"v1","addr":"172.21.15.17:6814","nonce":12462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":12462},{"type":"v1","addr":"172.21.15.17:6819","nonce":12462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":12462},{"type":"v1","addr":"172.21.15.17:6823","nonce":12462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":12462},{"type":"v1","addr":"172.21.15.17:6821","nonce":12462}]},"public_addr":"172.21.15.17:6814/12462","cluster_addr":"172.21.15.17:6819/12462","heartbeat_back_addr":"172.21.15.17:6823/12462","heartbeat_front_addr":"172.21.15.17:6821/12462","state":["exists","up"]},{"osd":2,"uuid":"e50c2a1e-7f9e-4803-b418-e50f3fc4c612","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":12461},{"type":"v1","addr":"172.21.15.17:6801","nonce":12461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":12461},{"type":"v1","addr":"172.21.15.17:6803","nonce":12461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":12461},{"type":"v1","addr":"172.21.15.17:6807","nonce":12461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":12461},{"type":"v1","addr":"172.21.15.17:6805","nonce":12461}]},"public_addr":"172.21.15.17:6801/12461","cluster_addr":"172.21.15.17:6803/12461","heartbeat_back_addr":"172.21.15.17:6807/12461","heartbeat_front_addr":"172.21.15.17:6805/12461","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.610648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.593456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.540948+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:13:59.026 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-02-08T07:13:59.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2021-02-08T07:13:59.292 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:13:59.292 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":13,"fsid":"cc72d099-b5fb-4f3a-9eb6-7ad41378824e","created":"2021-02-08T07:13:46.473218+0000","modified":"2021-02-08T07:13:58.553964+0000","last_up_change":"2021-02-08T07:13:52.511929+0000","last_in_change":"2021-02-08T07:13:52.511929+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-08T07:13:50.824449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-08T07:13:55.700003+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8765b53a-d24c-469b-9941-e8010b54fac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":12460},{"type":"v1","addr":"172.21.15.17:6809","nonce":12460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":12460},{"type":"v1","addr":"172.21.15.17:6812","nonce":12460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6817","nonce":12460},{"type":"v1","addr":"172.21.15.17:6818","nonce":12460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6813","nonce":12460},{"type":"v1","addr":"172.21.15.17:6815","nonce":12460}]},"public_addr":"172.21.15.17:6809/12460","cluster_addr":"172.21.15.17:6812/12460","heartbeat_back_addr":"172.21.15.17:6818/12460","heartbeat_front_addr":"172.21.15.17:6815/12460","state":["exists","up"]},{"osd":1,"uuid":"295ae8ad-356e-453c-8b91-63f53a1ee639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":12462},{"type":"v1","addr":"172.21.15.17:6814","nonce":12462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":12462},{"type":"v1","addr":"172.21.15.17:6819","nonce":12462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":12462},{"type":"v1","addr":"172.21.15.17:6823","nonce":12462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":12462},{"type":"v1","addr":"172.21.15.17:6821","nonce":12462}]},"public_addr":"172.21.15.17:6814/12462","cluster_addr":"172.21.15.17:6819/12462","heartbeat_back_addr":"172.21.15.17:6823/12462","heartbeat_front_addr":"172.21.15.17:6821/12462","state":["exists","up"]},{"osd":2,"uuid":"e50c2a1e-7f9e-4803-b418-e50f3fc4c612","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":12461},{"type":"v1","addr":"172.21.15.17:6801","nonce":12461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":12461},{"type":"v1","addr":"172.21.15.17:6803","nonce":12461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":12461},{"type":"v1","addr":"172.21.15.17:6807","nonce":12461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":12461},{"type":"v1","addr":"172.21.15.17:6805","nonce":12461}]},"public_addr":"172.21.15.17:6801/12461","cluster_addr":"172.21.15.17:6803/12461","heartbeat_back_addr":"172.21.15.17:6807/12461","heartbeat_front_addr":"172.21.15.17:6805/12461","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.610648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.593456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:13:50.540948+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:13:59.306 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2021-02-08T07:13:59.462 INFO:teuthology.orchestra.run.smithi017.stdout:34359738371 2021-02-08T07:13:59.463 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2021-02-08T07:13:59.629 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2021-02-08T07:13:59.631 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2021-02-08T07:13:59.797 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2021-02-08T07:13:59.798 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2021-02-08T07:14:00.080 INFO:teuthology.orchestra.run.smithi017.stdout:34359738370 2021-02-08T07:14:00.095 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2021-02-08T07:14:01.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2021-02-08T07:14:01.373 INFO:teuthology.orchestra.run.smithi017.stdout:34359738370 2021-02-08T07:14:01.392 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2021-02-08T07:14:02.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2021-02-08T07:14:02.678 INFO:teuthology.orchestra.run.smithi017.stdout:34359738371 2021-02-08T07:14:02.699 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2021-02-08T07:14:02.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2021-02-08T07:14:02.970 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2021-02-08T07:14:02.983 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2021-02-08T07:14:02.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2021-02-08T07:14:03.250 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2021-02-08T07:14:03.263 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2021-02-08T07:14:03.263 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-02-08T07:14:03.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2021-02-08T07:14:03.528 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:03.531 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2021-02-08T07:14:03.541 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-02-08T07:14:02.805997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.249968"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.572339+0000","last_change":"2021-02-08T07:13:57.568357+0000","last_active":"2021-02-08T07:13:57.572339+0000","last_peered":"2021-02-08T07:13:57.572339+0000","last_clean":"2021-02-08T07:13:57.572339+0000","last_became_active":"2021-02-08T07:13:57.567920+0000","last_became_peered":"2021-02-08T07:13:57.567920+0000","last_unstale":"2021-02-08T07:13:57.572339+0000","last_undegraded":"2021-02-08T07:13:57.572339+0000","last_fullsized":"2021-02-08T07:13:57.572339+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.572319+0000","last_change":"2021-02-08T07:13:57.568251+0000","last_active":"2021-02-08T07:13:57.572319+0000","last_peered":"2021-02-08T07:13:57.572319+0000","last_clean":"2021-02-08T07:13:57.572319+0000","last_became_active":"2021-02-08T07:13:57.567747+0000","last_became_peered":"2021-02-08T07:13:57.567747+0000","last_unstale":"2021-02-08T07:13:57.572319+0000","last_undegraded":"2021-02-08T07:13:57.572319+0000","last_fullsized":"2021-02-08T07:13:57.572319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.572352+0000","last_change":"2021-02-08T07:13:57.569194+0000","last_active":"2021-02-08T07:13:57.572352+0000","last_peered":"2021-02-08T07:13:57.572352+0000","last_clean":"2021-02-08T07:13:57.572352+0000","last_became_active":"2021-02-08T07:13:57.568773+0000","last_became_peered":"2021-02-08T07:13:57.568773+0000","last_unstale":"2021-02-08T07:13:57.572352+0000","last_undegraded":"2021-02-08T07:13:57.572352+0000","last_fullsized":"2021-02-08T07:13:57.572352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.565292+0000","last_change":"2021-02-08T07:13:57.565292+0000","last_active":"2021-02-08T07:13:57.565292+0000","last_peered":"2021-02-08T07:13:57.565292+0000","last_clean":"2021-02-08T07:13:57.565292+0000","last_became_active":"2021-02-08T07:13:57.564859+0000","last_became_peered":"2021-02-08T07:13:57.564859+0000","last_unstale":"2021-02-08T07:13:57.565292+0000","last_undegraded":"2021-02-08T07:13:57.565292+0000","last_fullsized":"2021-02-08T07:13:57.565292+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.557883+0000","last_change":"2021-02-08T07:13:54.504618+0000","last_active":"2021-02-08T07:13:57.557883+0000","last_peered":"2021-02-08T07:13:57.557883+0000","last_clean":"2021-02-08T07:13:57.557883+0000","last_became_active":"2021-02-08T07:13:54.503412+0000","last_became_peered":"2021-02-08T07:13:54.503412+0000","last_unstale":"2021-02-08T07:13:57.557883+0000","last_undegraded":"2021-02-08T07:13:57.557883+0000","last_fullsized":"2021-02-08T07:13:57.557883+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:51.501184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:51.501184+0000","last_clean_scrub_stamp":"2021-02-08T07:13:51.501184+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.567273+0000","last_change":"2021-02-08T07:13:57.562944+0000","last_active":"2021-02-08T07:13:57.567273+0000","last_peered":"2021-02-08T07:13:57.567273+0000","last_clean":"2021-02-08T07:13:57.567273+0000","last_became_active":"2021-02-08T07:13:57.562381+0000","last_became_peered":"2021-02-08T07:13:57.562381+0000","last_unstale":"2021-02-08T07:13:57.567273+0000","last_undegraded":"2021-02-08T07:13:57.567273+0000","last_fullsized":"2021-02-08T07:13:57.567273+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.565537+0000","last_change":"2021-02-08T07:13:57.565537+0000","last_active":"2021-02-08T07:13:57.565537+0000","last_peered":"2021-02-08T07:13:57.565537+0000","last_clean":"2021-02-08T07:13:57.565537+0000","last_became_active":"2021-02-08T07:13:57.564985+0000","last_became_peered":"2021-02-08T07:13:57.564985+0000","last_unstale":"2021-02-08T07:13:57.565537+0000","last_undegraded":"2021-02-08T07:13:57.565537+0000","last_fullsized":"2021-02-08T07:13:57.565537+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.567501+0000","last_change":"2021-02-08T07:13:57.567501+0000","last_active":"2021-02-08T07:13:57.567501+0000","last_peered":"2021-02-08T07:13:57.567501+0000","last_clean":"2021-02-08T07:13:57.567501+0000","last_became_active":"2021-02-08T07:13:57.567036+0000","last_became_peered":"2021-02-08T07:13:57.567036+0000","last_unstale":"2021-02-08T07:13:57.567501+0000","last_undegraded":"2021-02-08T07:13:57.567501+0000","last_fullsized":"2021-02-08T07:13:57.567501+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.572375+0000","last_change":"2021-02-08T07:13:57.569557+0000","last_active":"2021-02-08T07:13:57.572375+0000","last_peered":"2021-02-08T07:13:57.572375+0000","last_clean":"2021-02-08T07:13:57.572375+0000","last_became_active":"2021-02-08T07:13:57.569038+0000","last_became_peered":"2021-02-08T07:13:57.569038+0000","last_unstale":"2021-02-08T07:13:57.572375+0000","last_undegraded":"2021-02-08T07:13:57.572375+0000","last_fullsized":"2021-02-08T07:13:57.572375+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3780000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]}]}} 2021-02-08T07:14:03.542 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2021-02-08T07:14:03.806 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:03.809 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2021-02-08T07:14:03.820 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-02-08T07:14:02.805997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.249968"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.572339+0000","last_change":"2021-02-08T07:13:57.568357+0000","last_active":"2021-02-08T07:13:57.572339+0000","last_peered":"2021-02-08T07:13:57.572339+0000","last_clean":"2021-02-08T07:13:57.572339+0000","last_became_active":"2021-02-08T07:13:57.567920+0000","last_became_peered":"2021-02-08T07:13:57.567920+0000","last_unstale":"2021-02-08T07:13:57.572339+0000","last_undegraded":"2021-02-08T07:13:57.572339+0000","last_fullsized":"2021-02-08T07:13:57.572339+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.572319+0000","last_change":"2021-02-08T07:13:57.568251+0000","last_active":"2021-02-08T07:13:57.572319+0000","last_peered":"2021-02-08T07:13:57.572319+0000","last_clean":"2021-02-08T07:13:57.572319+0000","last_became_active":"2021-02-08T07:13:57.567747+0000","last_became_peered":"2021-02-08T07:13:57.567747+0000","last_unstale":"2021-02-08T07:13:57.572319+0000","last_undegraded":"2021-02-08T07:13:57.572319+0000","last_fullsized":"2021-02-08T07:13:57.572319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.572352+0000","last_change":"2021-02-08T07:13:57.569194+0000","last_active":"2021-02-08T07:13:57.572352+0000","last_peered":"2021-02-08T07:13:57.572352+0000","last_clean":"2021-02-08T07:13:57.572352+0000","last_became_active":"2021-02-08T07:13:57.568773+0000","last_became_peered":"2021-02-08T07:13:57.568773+0000","last_unstale":"2021-02-08T07:13:57.572352+0000","last_undegraded":"2021-02-08T07:13:57.572352+0000","last_fullsized":"2021-02-08T07:13:57.572352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.565292+0000","last_change":"2021-02-08T07:13:57.565292+0000","last_active":"2021-02-08T07:13:57.565292+0000","last_peered":"2021-02-08T07:13:57.565292+0000","last_clean":"2021-02-08T07:13:57.565292+0000","last_became_active":"2021-02-08T07:13:57.564859+0000","last_became_peered":"2021-02-08T07:13:57.564859+0000","last_unstale":"2021-02-08T07:13:57.565292+0000","last_undegraded":"2021-02-08T07:13:57.565292+0000","last_fullsized":"2021-02-08T07:13:57.565292+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.557883+0000","last_change":"2021-02-08T07:13:54.504618+0000","last_active":"2021-02-08T07:13:57.557883+0000","last_peered":"2021-02-08T07:13:57.557883+0000","last_clean":"2021-02-08T07:13:57.557883+0000","last_became_active":"2021-02-08T07:13:54.503412+0000","last_became_peered":"2021-02-08T07:13:54.503412+0000","last_unstale":"2021-02-08T07:13:57.557883+0000","last_undegraded":"2021-02-08T07:13:57.557883+0000","last_fullsized":"2021-02-08T07:13:57.557883+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:51.501184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:51.501184+0000","last_clean_scrub_stamp":"2021-02-08T07:13:51.501184+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.567273+0000","last_change":"2021-02-08T07:13:57.562944+0000","last_active":"2021-02-08T07:13:57.567273+0000","last_peered":"2021-02-08T07:13:57.567273+0000","last_clean":"2021-02-08T07:13:57.567273+0000","last_became_active":"2021-02-08T07:13:57.562381+0000","last_became_peered":"2021-02-08T07:13:57.562381+0000","last_unstale":"2021-02-08T07:13:57.567273+0000","last_undegraded":"2021-02-08T07:13:57.567273+0000","last_fullsized":"2021-02-08T07:13:57.567273+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.565537+0000","last_change":"2021-02-08T07:13:57.565537+0000","last_active":"2021-02-08T07:13:57.565537+0000","last_peered":"2021-02-08T07:13:57.565537+0000","last_clean":"2021-02-08T07:13:57.565537+0000","last_became_active":"2021-02-08T07:13:57.564985+0000","last_became_peered":"2021-02-08T07:13:57.564985+0000","last_unstale":"2021-02-08T07:13:57.565537+0000","last_undegraded":"2021-02-08T07:13:57.565537+0000","last_fullsized":"2021-02-08T07:13:57.565537+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.567501+0000","last_change":"2021-02-08T07:13:57.567501+0000","last_active":"2021-02-08T07:13:57.567501+0000","last_peered":"2021-02-08T07:13:57.567501+0000","last_clean":"2021-02-08T07:13:57.567501+0000","last_became_active":"2021-02-08T07:13:57.567036+0000","last_became_peered":"2021-02-08T07:13:57.567036+0000","last_unstale":"2021-02-08T07:13:57.567501+0000","last_undegraded":"2021-02-08T07:13:57.567501+0000","last_fullsized":"2021-02-08T07:13:57.567501+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:13:57.572375+0000","last_change":"2021-02-08T07:13:57.569557+0000","last_active":"2021-02-08T07:13:57.572375+0000","last_peered":"2021-02-08T07:13:57.572375+0000","last_clean":"2021-02-08T07:13:57.572375+0000","last_became_active":"2021-02-08T07:13:57.569038+0000","last_became_peered":"2021-02-08T07:13:57.569038+0000","last_unstale":"2021-02-08T07:13:57.572375+0000","last_undegraded":"2021-02-08T07:13:57.572375+0000","last_fullsized":"2021-02-08T07:13:57.572375+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:13:56.536455+0000","last_clean_scrub_stamp":"2021-02-08T07:13:56.536455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3780000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]}]}} 2021-02-08T07:14:03.821 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-02-08T07:14:03.821 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-02-08T07:14:03.821 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-02-08T07:14:03.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2021-02-08T07:14:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.145 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-08T07:14:04.146 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-02-08T07:14:04.146 INFO:teuthology.run_tasks:Running task ceph... 2021-02-08T07:14:04.161 INFO:tasks.ceph:Creating ceph cluster c2... 2021-02-08T07:14:04.161 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-08T07:14:04.162 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864577', 'branch': 'wip-yuri3-testing-2021-02-03-1046-octopus', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5864577', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2021-02-07_16:16:23-rgw-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2021-02-03-1046-octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '133532e25eccb46507b425783df68c050462a609', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '133532e25eccb46507b425783df68c050462a609'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuri3-testing-2021-02-03-1046-octopus', 'sha1': '133532e25eccb46507b425783df68c050462a609'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '133532e25eccb46507b425783df68c050462a609', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri3-testing-2021-02-03-1046-octopus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_133532e25eccb46507b425783df68c050462a609/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '133532e25eccb46507b425783df68c050462a609', 'targets': {'smithi017.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqsworecZ+NXvz4RgtnimxK4DRM14Ko8sQhTWWw6hOfRi9MqTDMncDlRIIOghbLxvFInRuOuHi/3+gpd77/1ytqJx7zC+eCamEq0Mc3j1fBxDInmIyIuctj/+rpolBZnsyXtPREEPTLrXPMufVqWHV8bxtYuVelT+PKbvTLdKuawY4l1oxsDSr5aJ4aHBA1FnAe15iFNNbQzsbbi0tObW1ag3znNRh5+XUIFwuJZJbhMe4DUFSdE2/Xc+2EQ6MmlvvylPN9vjVJ1hu0aihPEPK0HJbpbk/u3aC/hlKu48Z7mokBux4Nmt9U/Jr0p02cENBoxmE//NCAZApA8SHYCil', 'smithi110.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDT7W0N7K3cyUIC6+xOIqn6tGKNOr8qH8UlFgikfk6WJjcL5Dpz/7AZUsa109iARI6teTVv/APcQ0Q/Cy/eXRmD9EfjCc/7i52q0S0muTTdQtLeCwn6F7C47x0KZobHgG/LWZY99i/OsL29ycoraHNTeR8DOM2hS47MxIbkQVykN4+EjG1GNIltOL5s8yfy6M2L+WMALpBRJhL5H+7cU9B7Sn+SG5IJ/bqzbtsonVr2CCe0QRBHAEtbxw+kNVw8MEdP01VR9QB05RV059xowBtLhUUWFJJ5F9sy5T072VYD+x5AKKdt51TagnD0mjq3kffa6AOURoQo4fBI7l9Q2nRz'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '133532e25eccb46507b425783df68c050462a609', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '133532e25eccb46507b425783df68c050462a609', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '17c099650191e7aa47a7d9f502facca74f9b4d44', 'timestamp': '2021-02-07_16:16:23', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-08T07:14:04.162 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-08T07:14:04.164 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-08T07:14:04.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2021-02-08T07:14:04.211 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2021-02-08T07:14:04.229 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:14:04.230 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2021-02-08T07:14:04.276 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-08T07:14:04.276 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-08T07:14:04.325 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-08T07:14:04.325 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-08T07:14:04.325 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 341 Links: 1 2021-02-08T07:14:04.326 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-08T07:14:04.326 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2021-02-08 07:10:59.043548099 +0000 2021-02-08T07:14:04.326 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2021-02-08 07:10:58.919610101 +0000 2021-02-08T07:14:04.326 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2021-02-08 07:10:58.919610101 +0000 2021-02-08T07:14:04.326 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2021-02-08T07:14:04.335 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2021-02-08T07:14:04.335 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2021-02-08T07:14:04.335 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000307821 s, 1.7 MB/s 2021-02-08T07:14:04.340 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-08T07:14:04.389 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-08T07:14:04.389 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-08T07:14:04.390 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 348 Links: 1 2021-02-08T07:14:04.390 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-08T07:14:04.390 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2021-02-08 07:10:59.303418093 +0000 2021-02-08T07:14:04.390 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2021-02-08 07:10:59.183478095 +0000 2021-02-08T07:14:04.390 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2021-02-08 07:10:59.183478095 +0000 2021-02-08T07:14:04.390 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2021-02-08T07:14:04.398 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2021-02-08T07:14:04.398 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2021-02-08T07:14:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000272267 s, 1.9 MB/s 2021-02-08T07:14:04.403 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-08T07:14:04.453 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-08T07:14:04.453 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-08T07:14:04.454 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 354 Links: 1 2021-02-08T07:14:04.454 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-08T07:14:04.454 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2021-02-08 07:10:59.535302088 +0000 2021-02-08T07:14:04.454 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2021-02-08 07:10:59.411364090 +0000 2021-02-08T07:14:04.454 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2021-02-08 07:10:59.411364090 +0000 2021-02-08T07:14:04.454 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2021-02-08T07:14:04.463 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2021-02-08T07:14:04.463 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2021-02-08T07:14:04.463 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000225985 s, 2.3 MB/s 2021-02-08T07:14:04.468 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-08T07:14:04.517 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-08T07:14:04.517 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-08T07:14:04.517 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 360 Links: 1 2021-02-08T07:14:04.517 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-08T07:14:04.518 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2021-02-08 07:10:59.731204083 +0000 2021-02-08T07:14:04.518 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2021-02-08 07:10:59.635252085 +0000 2021-02-08T07:14:04.518 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2021-02-08 07:10:59.635252085 +0000 2021-02-08T07:14:04.518 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2021-02-08T07:14:04.527 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2021-02-08T07:14:04.527 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2021-02-08T07:14:04.527 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000230853 s, 2.2 MB/s 2021-02-08T07:14:04.530 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-08T07:14:04.531 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-08T07:14:04.531 INFO:tasks.ceph:Generating config... 2021-02-08T07:14:04.532 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-08T07:14:04.532 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-08T07:14:04.532 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-08T07:14:04.533 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-08T07:14:04.533 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-08T07:14:04.533 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-08T07:14:04.533 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-08T07:14:04.533 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-08T07:14:04.533 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-08T07:14:04.533 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-08T07:14:04.533 INFO:tasks.ceph:[client] setuser = ceph 2021-02-08T07:14:04.533 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-08T07:14:04.533 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-08T07:14:04.533 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-08T07:14:04.533 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-08T07:14:04.533 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-08T07:14:04.533 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-08T07:14:04.534 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-08T07:14:04.534 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-08T07:14:04.534 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-08T07:14:04.534 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-08T07:14:04.534 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-08T07:14:04.534 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-08T07:14:04.534 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-08T07:14:04.534 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-08T07:14:04.534 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-08T07:14:04.534 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-08T07:14:04.534 INFO:tasks.ceph:Setting up c2.mon.a... 2021-02-08T07:14:04.535 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-02-08T07:14:04.609 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/c2.keyring 2021-02-08T07:14:04.612 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-02-08T07:14:04.688 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-08T07:14:04.745 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.110')] 2021-02-08T07:14:04.745 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.110 --print /home/ubuntu/cephtest/c2.monmap 2021-02-08T07:14:04.817 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-02-08T07:14:04.817 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid 62f30fdf-394b-4c7d-ab35-c1e5185eefe5 2021-02-08T07:14:04.817 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2021-02-08T07:14:04.818 INFO:teuthology.orchestra.run.smithi110.stdout:fsid 62f30fdf-394b-4c7d-ab35-c1e5185eefe5 2021-02-08T07:14:04.818 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2021-02-08T07:14:04.817677+0000 2021-02-08T07:14:04.818 INFO:teuthology.orchestra.run.smithi110.stdout:created 2021-02-08T07:14:04.817677+0000 2021-02-08T07:14:04.818 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 0 (unknown) 2021-02-08T07:14:04.818 INFO:teuthology.orchestra.run.smithi110.stdout:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2021-02-08T07:14:04.818 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-02-08T07:14:04.820 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 62f30fdf-394b-4c7d-ab35-c1e5185eefe5... 2021-02-08T07:14:04.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-08T07:14:04.824 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-08T07:14:04.867 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2021-02-08T07:14:04.867 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2021-02-08T07:14:04.867 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-08T07:14:04.867 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2021-02-08T07:14:04.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.868 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2021-02-08T07:14:04.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.868 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2021-02-08T07:14:04.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.868 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2021-02-08T07:14:04.869 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2021-02-08T07:14:04.869 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.869 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2021-02-08T07:14:04.869 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2021-02-08T07:14:04.869 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.869 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-08T07:14:04.870 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2021-02-08T07:14:04.870 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.870 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2021-02-08T07:14:04.870 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-08T07:14:04.870 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.870 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2021-02-08T07:14:04.871 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2021-02-08T07:14:04.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2021-02-08T07:14:04.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2021-02-08T07:14:04.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2021-02-08T07:14:04.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2021-02-08T07:14:04.871 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2021-02-08T07:14:04.872 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2021-02-08T07:14:04.872 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-08T07:14:04.872 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-08T07:14:04.872 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.872 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-08T07:14:04.872 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.873 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2021-02-08T07:14:04.873 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.873 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2021-02-08T07:14:04.873 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.873 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2021-02-08T07:14:04.873 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2021-02-08T07:14:04.874 INFO:teuthology.orchestra.run.smithi017.stdout: mon health detail to clog = false 2021-02-08T07:14:04.874 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = 172.21.15.110 2021-02-08T07:14:04.874 INFO:teuthology.orchestra.run.smithi017.stdout: osd heartbeat grace = 40 2021-02-08T07:14:04.874 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = 62f30fdf-394b-4c7d-ab35-c1e5185eefe5 2021-02-08T07:14:04.874 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.874 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2021-02-08T07:14:04.875 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2021-02-08T07:14:04.875 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.875 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2021-02-08T07:14:04.875 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2021-02-08T07:14:04.875 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.875 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2021-02-08T07:14:04.875 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2021-02-08T07:14:04.876 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.876 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2021-02-08T07:14:04.876 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2021-02-08T07:14:04.876 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2021-02-08T07:14:04.876 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.876 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2021-02-08T07:14:04.877 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2021-02-08T07:14:04.877 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.877 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2021-02-08T07:14:04.877 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.877 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2021-02-08T07:14:04.877 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.877 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2021-02-08T07:14:04.878 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.878 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2021-02-08T07:14:04.878 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2021-02-08T07:14:04.878 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.878 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2021-02-08T07:14:04.878 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2021-02-08T07:14:04.879 INFO:teuthology.orchestra.run.smithi017.stdout: debug filestore = 20 2021-02-08T07:14:04.879 INFO:teuthology.orchestra.run.smithi017.stdout: debug journal = 20 2021-02-08T07:14:04.879 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 20 2021-02-08T07:14:04.879 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 25 2021-02-08T07:14:04.879 INFO:teuthology.orchestra.run.smithi017.stdout: osd fast shutdown = False 2021-02-08T07:14:04.879 INFO:teuthology.orchestra.run.smithi017.stdout: rocksdb delete range threshold = 0 2021-02-08T07:14:04.879 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.880 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2021-02-08T07:14:04.880 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2021-02-08T07:14:04.880 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2021-02-08T07:14:04.880 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2021-02-08T07:14:04.880 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2021-02-08T07:14:04.880 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2021-02-08T07:14:04.880 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2021-02-08T07:14:04.881 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2021-02-08T07:14:04.881 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.881 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2021-02-08T07:14:04.881 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2021-02-08T07:14:04.881 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2021-02-08T07:14:04.881 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2021-02-08T07:14:04.882 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2021-02-08T07:14:04.882 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2021-02-08T07:14:04.882 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2021-02-08T07:14:04.882 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2021-02-08T07:14:04.882 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2021-02-08T07:14:04.882 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2021-02-08T07:14:04.883 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2021-02-08T07:14:04.883 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd crush smoke test = False 2021-02-08T07:14:04.883 INFO:teuthology.orchestra.run.smithi017.stdout: 2021-02-08T07:14:04.883 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2021-02-08T07:14:04.883 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2021-02-08T07:14:04.883 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2021-02-08T07:14:04.883 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2021-02-08T07:14:04.884 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-08T07:14:04.884 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-08T07:14:04.884 INFO:teuthology.orchestra.run.smithi017.stdout: debug rgw = 20 2021-02-08T07:14:04.884 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt require ssl = False 2021-02-08T07:14:04.884 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms backend = testing 2021-02-08T07:14:04.884 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-08T07:14:04.884 INFO:teuthology.orchestra.run.smithi017.stdout: rgw curl low speed time = 300 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout: rgw data log num shards = 4 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout: rgw md log max shards = 4 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync log trim interval = 0 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync obj etag verify = True 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout: setgroup = ceph 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout: setuser = ceph 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10 2021-02-08T07:14:04.885 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2021-02-08T07:14:04.886 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 1000 2021-02-08T07:14:04.886 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2021-02-08T07:14:04.886 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10000 2021-02-08T07:14:04.897 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2021-02-08T07:14:04.898 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2021-02-08T07:14:04.898 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-08T07:14:04.899 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2021-02-08T07:14:04.899 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.900 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2021-02-08T07:14:04.900 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.900 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2021-02-08T07:14:04.900 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.900 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2021-02-08T07:14:04.900 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2021-02-08T07:14:04.900 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2021-02-08T07:14:04.901 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.902 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2021-02-08T07:14:04.902 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2021-02-08T07:14:04.902 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2021-02-08T07:14:04.902 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2021-02-08T07:14:04.902 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2021-02-08T07:14:04.903 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2021-02-08T07:14:04.904 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2021-02-08T07:14:04.904 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = 172.21.15.110 2021-02-08T07:14:04.904 INFO:teuthology.orchestra.run.smithi110.stdout: osd heartbeat grace = 40 2021-02-08T07:14:04.904 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 62f30fdf-394b-4c7d-ab35-c1e5185eefe5 2021-02-08T07:14:04.904 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.904 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2021-02-08T07:14:04.904 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2021-02-08T07:14:04.904 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2021-02-08T07:14:04.905 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2021-02-08T07:14:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: debug filestore = 20 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: debug journal = 20 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 20 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 25 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: osd fast shutdown = False 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: rocksdb delete range threshold = 0 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2021-02-08T07:14:04.907 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2021-02-08T07:14:04.908 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd crush smoke test = False 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: debug rgw = 20 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: rgw crypt require ssl = False 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: rgw crypt s3 kms backend = testing 2021-02-08T07:14:04.909 INFO:teuthology.orchestra.run.smithi110.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: rgw curl low speed time = 300 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: rgw data log num shards = 4 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: rgw md log max shards = 4 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: rgw sync log trim interval = 0 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: rgw sync obj etag verify = True 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: setgroup = ceph 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: setuser = ceph 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: osd_max_omap_entries_per_request = 10 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: osd_max_omap_entries_per_request = 1000 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2021-02-08T07:14:04.910 INFO:teuthology.orchestra.run.smithi110.stdout: osd_max_omap_entries_per_request = 10000 2021-02-08T07:14:04.911 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-02-08T07:14:04.912 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-02-08T07:14:05.012 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-08T07:14:05.096 DEBUG:teuthology.orchestra.remote:smithi110:/etc/ceph/c2.keyring is 206B 2021-02-08T07:14:05.174 DEBUG:teuthology.orchestra.remote:smithi110:/home/ubuntu/cephtest/c2.monmap is 179B 2021-02-08T07:14:05.216 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2021-02-08T07:14:05.217 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:14:05.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/c2.keyring 2021-02-08T07:14:05.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-08T07:14:05.245 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2021-02-08T07:14:05.245 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-08T07:14:05.297 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2021-02-08T07:14:05.297 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:14:05.297 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/c2.keyring 2021-02-08T07:14:05.297 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-08T07:14:05.325 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2021-02-08T07:14:05.325 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-08T07:14:05.427 INFO:tasks.ceph:Setting up mon nodes... 2021-02-08T07:14:05.427 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-08T07:14:05.428 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-02-08T07:14:05.516 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-02-08T07:14:05.518 INFO:tasks.ceph:Setting up mds nodes... 2021-02-08T07:14:05.518 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-08T07:14:05.518 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-02-08T07:14:05.595 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/c2.client.0.keyring 2021-02-08T07:14:05.608 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-02-08T07:14:05.686 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/c2.client.1.keyring 2021-02-08T07:14:05.697 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-08T07:14:05.697 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-08T07:14:05.697 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-02-08T07:14:05.755 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-08T07:14:05.755 INFO:tasks.ceph:role: c2.osd.0 2021-02-08T07:14:05.756 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2021-02-08T07:14:05.756 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-08T07:14:05.840 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-08T07:14:05.840 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-08T07:14:05.840 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-08T07:14:05.840 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-08T07:14:05.840 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2021-02-08T07:14:05.840 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-08T07:14:05.840 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-08T07:14:05.840 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-08T07:14:05.841 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-08T07:14:05.842 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2021-02-08T07:14:05.842 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-02-08T07:14:05.908 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-02-08T07:14:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2021-02-08T07:14:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-08T07:14:05.963 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-02-08T07:14:06.021 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-08T07:14:06.021 INFO:tasks.ceph:role: c2.osd.1 2021-02-08T07:14:06.021 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2021-02-08T07:14:06.021 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-08T07:14:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-08T07:14:06.103 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-08T07:14:06.103 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-08T07:14:06.103 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-08T07:14:06.103 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2021-02-08T07:14:06.103 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-08T07:14:06.103 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-08T07:14:06.103 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-08T07:14:06.103 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-08T07:14:06.104 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2021-02-08T07:14:06.105 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-02-08T07:14:06.165 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-02-08T07:14:06.217 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2021-02-08T07:14:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-08T07:14:06.219 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-02-08T07:14:06.277 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-08T07:14:06.277 INFO:tasks.ceph:role: c2.osd.2 2021-02-08T07:14:06.277 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2021-02-08T07:14:06.277 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-08T07:14:06.358 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-08T07:14:06.359 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-08T07:14:06.359 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-08T07:14:06.359 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-08T07:14:06.359 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2021-02-08T07:14:06.359 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-08T07:14:06.359 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-08T07:14:06.359 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-08T07:14:06.359 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-08T07:14:06.360 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2021-02-08T07:14:06.360 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-02-08T07:14:06.421 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-02-08T07:14:06.472 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2021-02-08T07:14:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-08T07:14:06.474 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-08T07:14:06.555 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:06.550+0000 7f436d9f0d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-02-08T07:14:06.555 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:06.550+0000 7f436d9f0d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-02-08T07:14:06.555 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:06.550+0000 7f436d9f0d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-08T07:14:06.556 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:06.550+0000 7f436d9f0d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-08T07:14:06.556 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:06.550+0000 7f436d9f0d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-08T07:14:06.556 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:06.554+0000 7f436d9f0d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-02-08T07:14:06.565 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:06.562+0000 7f436d9f0d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-02-08T07:14:08.457 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-02-08T07:14:08.514 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-08T07:14:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:08.590+0000 7f6140e7ad80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-02-08T07:14:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:08.590+0000 7f6140e7ad80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-02-08T07:14:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:08.590+0000 7f6140e7ad80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-08T07:14:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:08.590+0000 7f6140e7ad80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-08T07:14:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:08.590+0000 7f6140e7ad80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-08T07:14:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:08.590+0000 7f6140e7ad80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-02-08T07:14:08.604 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:08.602+0000 7f6140e7ad80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-02-08T07:14:10.497 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-02-08T07:14:10.556 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-08T07:14:10.636 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:10.634+0000 7f6ce6645d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-02-08T07:14:10.636 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:10.634+0000 7f6ce6645d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-02-08T07:14:10.636 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:10.634+0000 7f6ce6645d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-08T07:14:10.636 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:10.634+0000 7f6ce6645d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-08T07:14:10.636 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:10.634+0000 7f6ce6645d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-08T07:14:10.636 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:10.634+0000 7f6ce6645d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-02-08T07:14:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:10.646+0000 7f6ce6645d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-02-08T07:14:12.553 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-02-08T07:14:12.610 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-08T07:14:12.611 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2021-02-08T07:14:12.660 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.03xu2RMSIf 2021-02-08T07:14:12.661 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.03xu2RMSIf 2021-02-08T07:14:12.717 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.03xu2RMSIf 2021-02-08T07:14:12.822 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.03xu2RMSIf is 56B 2021-02-08T07:14:12.833 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.03xu2RMSIf 2021-02-08T07:14:12.839 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2021-02-08T07:14:12.888 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.ttTaiZdrW8 2021-02-08T07:14:12.889 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.ttTaiZdrW8 2021-02-08T07:14:12.945 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.ttTaiZdrW8 2021-02-08T07:14:13.051 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.ttTaiZdrW8 is 56B 2021-02-08T07:14:13.100 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.ttTaiZdrW8 2021-02-08T07:14:13.106 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2021-02-08T07:14:13.152 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.EGZ9vIzhyd 2021-02-08T07:14:13.152 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.EGZ9vIzhyd 2021-02-08T07:14:13.209 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.EGZ9vIzhyd 2021-02-08T07:14:13.314 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.EGZ9vIzhyd is 56B 2021-02-08T07:14:13.325 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.EGZ9vIzhyd 2021-02-08T07:14:13.331 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2021-02-08T07:14:13.380 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.5FO6wYCFg0 2021-02-08T07:14:13.380 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.5FO6wYCFg0 2021-02-08T07:14:13.437 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.5FO6wYCFg0 2021-02-08T07:14:13.542 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.5FO6wYCFg0 is 56B 2021-02-08T07:14:13.554 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.5FO6wYCFg0 2021-02-08T07:14:13.614 DEBUG:teuthology.orchestra.remote:smithi110:/etc/ceph/c2.client.0.keyring is 59B 2021-02-08T07:14:13.634 DEBUG:teuthology.orchestra.remote:smithi110:/etc/ceph/c2.client.1.keyring is 59B 2021-02-08T07:14:13.645 INFO:tasks.ceph:Adding keys to all mons... 2021-02-08T07:14:13.645 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/c2.keyring 2021-02-08T07:14:13.657 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2021-02-08T07:14:13.657 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC95CBga0TNHhAAUAKqPuAIBAmo7uCjUQTTkg== 2021-02-08T07:14:13.657 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2021-02-08T07:14:13.657 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC+5CBg8NEaIRAAOCh1nDrVapktpERsMAjC7w== 2021-02-08T07:14:13.657 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2021-02-08T07:14:13.657 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDA5CBg7w9bIxAAoa19Hl/J1hOOnPDNWug5QA== 2021-02-08T07:14:13.657 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2021-02-08T07:14:13.657 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDC5CBg6S/wJRAAH7vZahMQx5n/RcY/n8e5Ww== 2021-02-08T07:14:13.657 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2021-02-08T07:14:13.658 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC95CBgoeOOIxAAO0Pd0V5x4DCmBcqxJnYihg== 2021-02-08T07:14:13.658 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2021-02-08T07:14:13.658 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC95CBgyvntKBAAuhE5pr5jUnTeiLZzkkr8GQ== 2021-02-08T07:14:13.659 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-08T07:14:13.744 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-08T07:14:13.828 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-08T07:14:13.912 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-08T07:14:13.996 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-08T07:14:14.085 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-08T07:14:14.126 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-08T07:14:14.126 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-02-08T07:14:14.179 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-02-08T07:14:14.295 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-02-08T07:14:14.310 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-02-08T07:14:14.356 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-02-08T07:14:14.356 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-02-08T07:14:14.356 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-02-08T07:14:14.400 INFO:tasks.ceph.c2.mon.a:Started 2021-02-08T07:14:14.400 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-02-08T07:14:14.400 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-02-08T07:14:14.400 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-02-08T07:14:14.402 INFO:tasks.ceph.c2.mgr.x:Started 2021-02-08T07:14:14.403 INFO:tasks.ceph:Setting crush tunables to default 2021-02-08T07:14:14.403 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster c2 osd crush tunables default 2021-02-08T07:14:14.741 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2021-02-08T07:14:14.751 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-02-08T07:14:14.752 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2021-02-08T07:14:14.756 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.6HO0zxiijI 2021-02-08T07:14:14.756 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.6HO0zxiijI 2021-02-08T07:14:14.817 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.6HO0zxiijI 2021-02-08T07:14:14.921 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.6HO0zxiijI is 37B 2021-02-08T07:14:14.933 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.6HO0zxiijI 2021-02-08T07:14:14.938 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2021-02-08T07:14:14.984 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.1IrKeOJmaL 2021-02-08T07:14:14.984 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.1IrKeOJmaL 2021-02-08T07:14:15.041 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.1IrKeOJmaL 2021-02-08T07:14:15.147 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.1IrKeOJmaL is 37B 2021-02-08T07:14:15.196 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.1IrKeOJmaL 2021-02-08T07:14:15.203 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2021-02-08T07:14:15.252 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.TyDQ1Q9jKs 2021-02-08T07:14:15.252 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.TyDQ1Q9jKs 2021-02-08T07:14:15.309 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.TyDQ1Q9jKs 2021-02-08T07:14:15.414 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.TyDQ1Q9jKs is 37B 2021-02-08T07:14:15.424 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.TyDQ1Q9jKs 2021-02-08T07:14:15.429 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster c2 osd new a52d251a-ae22-4caf-a304-fe26272739cb 0 2021-02-08T07:14:15.683 INFO:teuthology.orchestra.run.smithi110.stdout:0 2021-02-08T07:14:15.692 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster c2 osd new e3fc3ebc-4762-4438-ae30-cda7f9e60f5e 1 2021-02-08T07:14:15.912 INFO:teuthology.orchestra.run.smithi110.stdout:1 2021-02-08T07:14:15.922 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster c2 osd new 4e344593-be83-4873-ab68-bae5ff853f12 2 2021-02-08T07:14:16.125 INFO:teuthology.orchestra.run.smithi110.stdout:2 2021-02-08T07:14:16.137 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-02-08T07:14:16.137 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-02-08T07:14:16.140 INFO:tasks.ceph.c2.osd.0:Started 2021-02-08T07:14:16.140 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-02-08T07:14:16.140 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-02-08T07:14:16.143 INFO:tasks.ceph.c2.osd.1:Started 2021-02-08T07:14:16.143 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-02-08T07:14:16.143 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-02-08T07:14:16.147 INFO:tasks.ceph.c2.osd.2:Started 2021-02-08T07:14:16.148 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-08T07:14:16.451 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-08T07:14:16.516 INFO:tasks.ceph.c2.osd.0.smithi110.stderr:2021-02-08T07:14:16.514+0000 7f38f8c27d80 -1 Falling back to public interface 2021-02-08T07:14:16.621 INFO:teuthology.misc.health.smithi110.stdout: 2021-02-08T07:14:16.621 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":5,"fsid":"62f30fdf-394b-4c7d-ab35-c1e5185eefe5","created":"2021-02-08T07:14:14.511073+0000","modified":"2021-02-08T07:14:16.122488+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"a52d251a-ae22-4caf-a304-fe26272739cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e3fc3ebc-4762-4438-ae30-cda7f9e60f5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e344593-be83-4873-ab68-bae5ff853f12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:14:16.631 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-08T07:14:16.744 INFO:tasks.ceph.c2.osd.1.smithi110.stderr:2021-02-08T07:14:16.742+0000 7f0e3c0bbd80 -1 Falling back to public interface 2021-02-08T07:14:16.756 INFO:tasks.ceph.c2.osd.2.smithi110.stderr:2021-02-08T07:14:16.754+0000 7fc838253d80 -1 Falling back to public interface 2021-02-08T07:14:17.105 INFO:tasks.ceph.c2.osd.0.smithi110.stderr:2021-02-08T07:14:17.102+0000 7f38f8c27d80 -1 osd.0 0 log_to_monitors {default=true} 2021-02-08T07:14:17.350 INFO:tasks.ceph.c2.osd.1.smithi110.stderr:2021-02-08T07:14:17.346+0000 7f0e3c0bbd80 -1 osd.1 0 log_to_monitors {default=true} 2021-02-08T07:14:17.389 INFO:tasks.ceph.c2.osd.2.smithi110.stderr:2021-02-08T07:14:17.386+0000 7fc838253d80 -1 osd.2 0 log_to_monitors {default=true} 2021-02-08T07:14:18.538 INFO:tasks.ceph.c2.osd.1.smithi110.stderr:2021-02-08T07:14:18.534+0000 7f0e25b07700 -1 osd.1 0 waiting for initial osdmap 2021-02-08T07:14:18.539 INFO:tasks.ceph.c2.osd.0.smithi110.stderr:2021-02-08T07:14:18.534+0000 7f38e2673700 -1 osd.0 0 waiting for initial osdmap 2021-02-08T07:14:18.539 INFO:tasks.ceph.c2.osd.2.smithi110.stderr:2021-02-08T07:14:18.538+0000 7fc821c9f700 -1 osd.2 0 waiting for initial osdmap 2021-02-08T07:14:22.934 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-08T07:14:23.195 INFO:teuthology.misc.health.smithi110.stdout: 2021-02-08T07:14:23.195 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"62f30fdf-394b-4c7d-ab35-c1e5185eefe5","created":"2021-02-08T07:14:14.511073+0000","modified":"2021-02-08T07:14:21.548769+0000","last_up_change":"2021-02-08T07:14:19.534093+0000","last_in_change":"2021-02-08T07:14:19.534093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-08T07:14:18.417585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"a52d251a-ae22-4caf-a304-fe26272739cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":12509},{"type":"v1","addr":"172.21.15.110:6801","nonce":12509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":12509},{"type":"v1","addr":"172.21.15.110:6803","nonce":12509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":12509},{"type":"v1","addr":"172.21.15.110:6807","nonce":12509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":12509},{"type":"v1","addr":"172.21.15.110:6805","nonce":12509}]},"public_addr":"172.21.15.110:6801/12509","cluster_addr":"172.21.15.110:6803/12509","heartbeat_back_addr":"172.21.15.110:6807/12509","heartbeat_front_addr":"172.21.15.110:6805/12509","state":["exists","up"]},{"osd":1,"uuid":"e3fc3ebc-4762-4438-ae30-cda7f9e60f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":12510},{"type":"v1","addr":"172.21.15.110:6809","nonce":12510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":12510},{"type":"v1","addr":"172.21.15.110:6811","nonce":12510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":12510},{"type":"v1","addr":"172.21.15.110:6815","nonce":12510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":12510},{"type":"v1","addr":"172.21.15.110:6813","nonce":12510}]},"public_addr":"172.21.15.110:6809/12510","cluster_addr":"172.21.15.110:6811/12510","heartbeat_back_addr":"172.21.15.110:6815/12510","heartbeat_front_addr":"172.21.15.110:6813/12510","state":["exists","up"]},{"osd":2,"uuid":"4e344593-be83-4873-ab68-bae5ff853f12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":12519},{"type":"v1","addr":"172.21.15.110:6817","nonce":12519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":12519},{"type":"v1","addr":"172.21.15.110:6819","nonce":12519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":12519},{"type":"v1","addr":"172.21.15.110:6823","nonce":12519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":12519},{"type":"v1","addr":"172.21.15.110:6821","nonce":12519}]},"public_addr":"172.21.15.110:6817/12519","cluster_addr":"172.21.15.110:6819/12519","heartbeat_back_addr":"172.21.15.110:6823/12519","heartbeat_front_addr":"172.21.15.110:6821/12519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.101084+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.310235+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.378210+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:14:23.207 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-08T07:14:23.207 INFO:tasks.ceph:Creating RBD pool 2021-02-08T07:14:23.208 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster c2 osd pool create rbd 8 2021-02-08T07:14:23.576 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'rbd' created 2021-02-08T07:14:23.589 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-08T07:14:24.586 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-08T07:14:24.598 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-02-08T07:14:24.599 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2021-02-08T07:14:24.600 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-08T07:14:24.862 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:24.862 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":12,"fsid":"62f30fdf-394b-4c7d-ab35-c1e5185eefe5","created":"2021-02-08T07:14:14.511073+0000","modified":"2021-02-08T07:14:24.565452+0000","last_up_change":"2021-02-08T07:14:19.534093+0000","last_in_change":"2021-02-08T07:14:19.534093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-08T07:14:18.417585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-08T07:14:23.478079+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a52d251a-ae22-4caf-a304-fe26272739cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":12509},{"type":"v1","addr":"172.21.15.110:6801","nonce":12509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":12509},{"type":"v1","addr":"172.21.15.110:6803","nonce":12509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":12509},{"type":"v1","addr":"172.21.15.110:6807","nonce":12509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":12509},{"type":"v1","addr":"172.21.15.110:6805","nonce":12509}]},"public_addr":"172.21.15.110:6801/12509","cluster_addr":"172.21.15.110:6803/12509","heartbeat_back_addr":"172.21.15.110:6807/12509","heartbeat_front_addr":"172.21.15.110:6805/12509","state":["exists","up"]},{"osd":1,"uuid":"e3fc3ebc-4762-4438-ae30-cda7f9e60f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":12510},{"type":"v1","addr":"172.21.15.110:6809","nonce":12510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":12510},{"type":"v1","addr":"172.21.15.110:6811","nonce":12510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":12510},{"type":"v1","addr":"172.21.15.110:6815","nonce":12510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":12510},{"type":"v1","addr":"172.21.15.110:6813","nonce":12510}]},"public_addr":"172.21.15.110:6809/12510","cluster_addr":"172.21.15.110:6811/12510","heartbeat_back_addr":"172.21.15.110:6815/12510","heartbeat_front_addr":"172.21.15.110:6813/12510","state":["exists","up"]},{"osd":2,"uuid":"4e344593-be83-4873-ab68-bae5ff853f12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":12519},{"type":"v1","addr":"172.21.15.110:6817","nonce":12519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":12519},{"type":"v1","addr":"172.21.15.110:6819","nonce":12519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":12519},{"type":"v1","addr":"172.21.15.110:6823","nonce":12519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":12519},{"type":"v1","addr":"172.21.15.110:6821","nonce":12519}]},"public_addr":"172.21.15.110:6817/12519","cluster_addr":"172.21.15.110:6819/12519","heartbeat_back_addr":"172.21.15.110:6823/12519","heartbeat_front_addr":"172.21.15.110:6821/12519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.101084+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.310235+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.378210+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:14:24.875 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-08T07:14:18.417585+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-08T07:14:23.478079+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-08T07:14:24.875 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2021-02-08T07:14:25.138 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2021-02-08T07:14:25.156 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2021-02-08T07:14:25.426 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2021-02-08T07:14:25.439 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-02-08T07:14:25.439 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-02-08T07:14:25.440 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2021-02-08T07:14:25.743 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:25.755 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":12300},{"type":"v1","addr":"172.21.15.110:6825","nonce":12300}]},"active_addr":"172.21.15.110:6825/12300","active_change":"2021-02-08T07:14:18.396951+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":3462171815}]},{"addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":4172433828}]},{"addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":238032526}]}]}} 2021-02-08T07:14:25.757 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-02-08T07:14:25.757 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-02-08T07:14:25.757 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2021-02-08T07:14:26.017 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:27.386 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":13,"fsid":"62f30fdf-394b-4c7d-ab35-c1e5185eefe5","created":"2021-02-08T07:14:14.511073+0000","modified":"2021-02-08T07:14:25.570084+0000","last_up_change":"2021-02-08T07:14:19.534093+0000","last_in_change":"2021-02-08T07:14:19.534093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-08T07:14:18.417585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-08T07:14:23.478079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a52d251a-ae22-4caf-a304-fe26272739cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":12509},{"type":"v1","addr":"172.21.15.110:6801","nonce":12509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":12509},{"type":"v1","addr":"172.21.15.110:6803","nonce":12509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":12509},{"type":"v1","addr":"172.21.15.110:6807","nonce":12509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":12509},{"type":"v1","addr":"172.21.15.110:6805","nonce":12509}]},"public_addr":"172.21.15.110:6801/12509","cluster_addr":"172.21.15.110:6803/12509","heartbeat_back_addr":"172.21.15.110:6807/12509","heartbeat_front_addr":"172.21.15.110:6805/12509","state":["exists","up"]},{"osd":1,"uuid":"e3fc3ebc-4762-4438-ae30-cda7f9e60f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":12510},{"type":"v1","addr":"172.21.15.110:6809","nonce":12510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":12510},{"type":"v1","addr":"172.21.15.110:6811","nonce":12510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":12510},{"type":"v1","addr":"172.21.15.110:6815","nonce":12510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":12510},{"type":"v1","addr":"172.21.15.110:6813","nonce":12510}]},"public_addr":"172.21.15.110:6809/12510","cluster_addr":"172.21.15.110:6811/12510","heartbeat_back_addr":"172.21.15.110:6815/12510","heartbeat_front_addr":"172.21.15.110:6813/12510","state":["exists","up"]},{"osd":2,"uuid":"4e344593-be83-4873-ab68-bae5ff853f12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":12519},{"type":"v1","addr":"172.21.15.110:6817","nonce":12519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":12519},{"type":"v1","addr":"172.21.15.110:6819","nonce":12519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":12519},{"type":"v1","addr":"172.21.15.110:6823","nonce":12519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":12519},{"type":"v1","addr":"172.21.15.110:6821","nonce":12519}]},"public_addr":"172.21.15.110:6817/12519","cluster_addr":"172.21.15.110:6819/12519","heartbeat_back_addr":"172.21.15.110:6823/12519","heartbeat_front_addr":"172.21.15.110:6821/12519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.101084+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.310235+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.378210+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:14:27.387 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-02-08T07:14:27.387 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2021-02-08T07:14:27.661 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:27.662 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":13,"fsid":"62f30fdf-394b-4c7d-ab35-c1e5185eefe5","created":"2021-02-08T07:14:14.511073+0000","modified":"2021-02-08T07:14:25.570084+0000","last_up_change":"2021-02-08T07:14:19.534093+0000","last_in_change":"2021-02-08T07:14:19.534093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-08T07:14:18.417585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-08T07:14:23.478079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a52d251a-ae22-4caf-a304-fe26272739cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":12509},{"type":"v1","addr":"172.21.15.110:6801","nonce":12509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":12509},{"type":"v1","addr":"172.21.15.110:6803","nonce":12509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":12509},{"type":"v1","addr":"172.21.15.110:6807","nonce":12509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":12509},{"type":"v1","addr":"172.21.15.110:6805","nonce":12509}]},"public_addr":"172.21.15.110:6801/12509","cluster_addr":"172.21.15.110:6803/12509","heartbeat_back_addr":"172.21.15.110:6807/12509","heartbeat_front_addr":"172.21.15.110:6805/12509","state":["exists","up"]},{"osd":1,"uuid":"e3fc3ebc-4762-4438-ae30-cda7f9e60f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":12510},{"type":"v1","addr":"172.21.15.110:6809","nonce":12510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":12510},{"type":"v1","addr":"172.21.15.110:6811","nonce":12510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":12510},{"type":"v1","addr":"172.21.15.110:6815","nonce":12510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":12510},{"type":"v1","addr":"172.21.15.110:6813","nonce":12510}]},"public_addr":"172.21.15.110:6809/12510","cluster_addr":"172.21.15.110:6811/12510","heartbeat_back_addr":"172.21.15.110:6815/12510","heartbeat_front_addr":"172.21.15.110:6813/12510","state":["exists","up"]},{"osd":2,"uuid":"4e344593-be83-4873-ab68-bae5ff853f12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":12519},{"type":"v1","addr":"172.21.15.110:6817","nonce":12519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":12519},{"type":"v1","addr":"172.21.15.110:6819","nonce":12519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":12519},{"type":"v1","addr":"172.21.15.110:6823","nonce":12519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":12519},{"type":"v1","addr":"172.21.15.110:6821","nonce":12519}]},"public_addr":"172.21.15.110:6817/12519","cluster_addr":"172.21.15.110:6819/12519","heartbeat_back_addr":"172.21.15.110:6823/12519","heartbeat_front_addr":"172.21.15.110:6821/12519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.101084+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.310235+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-02-08T07:14:18.378210+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-08T07:14:27.675 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2021-02-08T07:14:27.825 INFO:teuthology.orchestra.run.smithi110.stdout:34359738371 2021-02-08T07:14:27.825 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2021-02-08T07:14:27.983 INFO:teuthology.orchestra.run.smithi110.stdout:34359738372 2021-02-08T07:14:27.983 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2021-02-08T07:14:28.141 INFO:teuthology.orchestra.run.smithi110.stdout:34359738372 2021-02-08T07:14:28.141 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2021-02-08T07:14:28.400 INFO:teuthology.orchestra.run.smithi110.stdout:34359738370 2021-02-08T07:14:28.413 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2021-02-08T07:14:29.414 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2021-02-08T07:14:29.696 INFO:teuthology.orchestra.run.smithi110.stdout:34359738371 2021-02-08T07:14:29.708 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2021-02-08T07:14:29.708 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2021-02-08T07:14:29.972 INFO:teuthology.orchestra.run.smithi110.stdout:34359738372 2021-02-08T07:14:29.991 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2021-02-08T07:14:29.991 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2021-02-08T07:14:30.250 INFO:teuthology.orchestra.run.smithi110.stdout:34359738372 2021-02-08T07:14:30.263 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2021-02-08T07:14:30.263 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-02-08T07:14:30.264 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2021-02-08T07:14:30.522 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:30.525 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2021-02-08T07:14:30.536 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-02-08T07:14:30.407152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.835047"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.587489+0000","last_change":"2021-02-08T07:14:24.587031+0000","last_active":"2021-02-08T07:14:24.587489+0000","last_peered":"2021-02-08T07:14:24.587489+0000","last_clean":"2021-02-08T07:14:24.587489+0000","last_became_active":"2021-02-08T07:14:24.586662+0000","last_became_peered":"2021-02-08T07:14:24.586662+0000","last_unstale":"2021-02-08T07:14:24.587489+0000","last_undegraded":"2021-02-08T07:14:24.587489+0000","last_fullsized":"2021-02-08T07:14:24.587489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.586190+0000","last_change":"2021-02-08T07:14:24.585802+0000","last_active":"2021-02-08T07:14:24.586190+0000","last_peered":"2021-02-08T07:14:24.586190+0000","last_clean":"2021-02-08T07:14:24.586190+0000","last_became_active":"2021-02-08T07:14:24.585381+0000","last_became_peered":"2021-02-08T07:14:24.585381+0000","last_unstale":"2021-02-08T07:14:24.586190+0000","last_undegraded":"2021-02-08T07:14:24.586190+0000","last_fullsized":"2021-02-08T07:14:24.586190+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.584840+0000","last_change":"2021-02-08T07:14:24.584207+0000","last_active":"2021-02-08T07:14:24.584840+0000","last_peered":"2021-02-08T07:14:24.584840+0000","last_clean":"2021-02-08T07:14:24.584840+0000","last_became_active":"2021-02-08T07:14:24.583737+0000","last_became_peered":"2021-02-08T07:14:24.583737+0000","last_unstale":"2021-02-08T07:14:24.584840+0000","last_undegraded":"2021-02-08T07:14:24.584840+0000","last_fullsized":"2021-02-08T07:14:24.584840+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.586299+0000","last_change":"2021-02-08T07:14:24.585472+0000","last_active":"2021-02-08T07:14:24.586299+0000","last_peered":"2021-02-08T07:14:24.586299+0000","last_clean":"2021-02-08T07:14:24.586299+0000","last_became_active":"2021-02-08T07:14:24.584814+0000","last_became_peered":"2021-02-08T07:14:24.584814+0000","last_unstale":"2021-02-08T07:14:24.586299+0000","last_undegraded":"2021-02-08T07:14:24.586299+0000","last_fullsized":"2021-02-08T07:14:24.586299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.581496+0000","last_change":"2021-02-08T07:14:21.114785+0000","last_active":"2021-02-08T07:14:24.581496+0000","last_peered":"2021-02-08T07:14:24.581496+0000","last_clean":"2021-02-08T07:14:24.581496+0000","last_became_active":"2021-02-08T07:14:21.113664+0000","last_became_peered":"2021-02-08T07:14:21.113664+0000","last_unstale":"2021-02-08T07:14:24.581496+0000","last_undegraded":"2021-02-08T07:14:24.581496+0000","last_fullsized":"2021-02-08T07:14:24.581496+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:18.526829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:18.526829+0000","last_clean_scrub_stamp":"2021-02-08T07:14:18.526829+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.588682+0000","last_change":"2021-02-08T07:14:24.588098+0000","last_active":"2021-02-08T07:14:24.588682+0000","last_peered":"2021-02-08T07:14:24.588682+0000","last_clean":"2021-02-08T07:14:24.588682+0000","last_became_active":"2021-02-08T07:14:24.587680+0000","last_became_peered":"2021-02-08T07:14:24.587680+0000","last_unstale":"2021-02-08T07:14:24.588682+0000","last_undegraded":"2021-02-08T07:14:24.588682+0000","last_fullsized":"2021-02-08T07:14:24.588682+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.586284+0000","last_change":"2021-02-08T07:14:24.585349+0000","last_active":"2021-02-08T07:14:24.586284+0000","last_peered":"2021-02-08T07:14:24.586284+0000","last_clean":"2021-02-08T07:14:24.586284+0000","last_became_active":"2021-02-08T07:14:24.584782+0000","last_became_peered":"2021-02-08T07:14:24.584782+0000","last_unstale":"2021-02-08T07:14:24.586284+0000","last_undegraded":"2021-02-08T07:14:24.586284+0000","last_fullsized":"2021-02-08T07:14:24.586284+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.586093+0000","last_change":"2021-02-08T07:14:24.585614+0000","last_active":"2021-02-08T07:14:24.586093+0000","last_peered":"2021-02-08T07:14:24.586093+0000","last_clean":"2021-02-08T07:14:24.586093+0000","last_became_active":"2021-02-08T07:14:24.585020+0000","last_became_peered":"2021-02-08T07:14:24.585020+0000","last_unstale":"2021-02-08T07:14:24.586093+0000","last_undegraded":"2021-02-08T07:14:24.586093+0000","last_fullsized":"2021-02-08T07:14:24.586093+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.587468+0000","last_change":"2021-02-08T07:14:24.586814+0000","last_active":"2021-02-08T07:14:24.587468+0000","last_peered":"2021-02-08T07:14:24.587468+0000","last_clean":"2021-02-08T07:14:24.587468+0000","last_became_active":"2021-02-08T07:14:24.586427+0000","last_became_peered":"2021-02-08T07:14:24.586427+0000","last_unstale":"2021-02-08T07:14:24.587468+0000","last_undegraded":"2021-02-08T07:14:24.587468+0000","last_fullsized":"2021-02-08T07:14:24.587468+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]}]}]}} 2021-02-08T07:14:30.536 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2021-02-08T07:14:30.795 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2021-02-08T07:14:30.808 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-02-08T07:14:30.407152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.835047"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.587489+0000","last_change":"2021-02-08T07:14:24.587031+0000","last_active":"2021-02-08T07:14:24.587489+0000","last_peered":"2021-02-08T07:14:24.587489+0000","last_clean":"2021-02-08T07:14:24.587489+0000","last_became_active":"2021-02-08T07:14:24.586662+0000","last_became_peered":"2021-02-08T07:14:24.586662+0000","last_unstale":"2021-02-08T07:14:24.587489+0000","last_undegraded":"2021-02-08T07:14:24.587489+0000","last_fullsized":"2021-02-08T07:14:24.587489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.586190+0000","last_change":"2021-02-08T07:14:24.585802+0000","last_active":"2021-02-08T07:14:24.586190+0000","last_peered":"2021-02-08T07:14:24.586190+0000","last_clean":"2021-02-08T07:14:24.586190+0000","last_became_active":"2021-02-08T07:14:24.585381+0000","last_became_peered":"2021-02-08T07:14:24.585381+0000","last_unstale":"2021-02-08T07:14:24.586190+0000","last_undegraded":"2021-02-08T07:14:24.586190+0000","last_fullsized":"2021-02-08T07:14:24.586190+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.584840+0000","last_change":"2021-02-08T07:14:24.584207+0000","last_active":"2021-02-08T07:14:24.584840+0000","last_peered":"2021-02-08T07:14:24.584840+0000","last_clean":"2021-02-08T07:14:24.584840+0000","last_became_active":"2021-02-08T07:14:24.583737+0000","last_became_peered":"2021-02-08T07:14:24.583737+0000","last_unstale":"2021-02-08T07:14:24.584840+0000","last_undegraded":"2021-02-08T07:14:24.584840+0000","last_fullsized":"2021-02-08T07:14:24.584840+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.586299+0000","last_change":"2021-02-08T07:14:24.585472+0000","last_active":"2021-02-08T07:14:24.586299+0000","last_peered":"2021-02-08T07:14:24.586299+0000","last_clean":"2021-02-08T07:14:24.586299+0000","last_became_active":"2021-02-08T07:14:24.584814+0000","last_became_peered":"2021-02-08T07:14:24.584814+0000","last_unstale":"2021-02-08T07:14:24.586299+0000","last_undegraded":"2021-02-08T07:14:24.586299+0000","last_fullsized":"2021-02-08T07:14:24.586299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.581496+0000","last_change":"2021-02-08T07:14:21.114785+0000","last_active":"2021-02-08T07:14:24.581496+0000","last_peered":"2021-02-08T07:14:24.581496+0000","last_clean":"2021-02-08T07:14:24.581496+0000","last_became_active":"2021-02-08T07:14:21.113664+0000","last_became_peered":"2021-02-08T07:14:21.113664+0000","last_unstale":"2021-02-08T07:14:24.581496+0000","last_undegraded":"2021-02-08T07:14:24.581496+0000","last_fullsized":"2021-02-08T07:14:24.581496+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:18.526829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:18.526829+0000","last_clean_scrub_stamp":"2021-02-08T07:14:18.526829+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.588682+0000","last_change":"2021-02-08T07:14:24.588098+0000","last_active":"2021-02-08T07:14:24.588682+0000","last_peered":"2021-02-08T07:14:24.588682+0000","last_clean":"2021-02-08T07:14:24.588682+0000","last_became_active":"2021-02-08T07:14:24.587680+0000","last_became_peered":"2021-02-08T07:14:24.587680+0000","last_unstale":"2021-02-08T07:14:24.588682+0000","last_undegraded":"2021-02-08T07:14:24.588682+0000","last_fullsized":"2021-02-08T07:14:24.588682+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.586284+0000","last_change":"2021-02-08T07:14:24.585349+0000","last_active":"2021-02-08T07:14:24.586284+0000","last_peered":"2021-02-08T07:14:24.586284+0000","last_clean":"2021-02-08T07:14:24.586284+0000","last_became_active":"2021-02-08T07:14:24.584782+0000","last_became_peered":"2021-02-08T07:14:24.584782+0000","last_unstale":"2021-02-08T07:14:24.586284+0000","last_undegraded":"2021-02-08T07:14:24.586284+0000","last_fullsized":"2021-02-08T07:14:24.586284+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.586093+0000","last_change":"2021-02-08T07:14:24.585614+0000","last_active":"2021-02-08T07:14:24.586093+0000","last_peered":"2021-02-08T07:14:24.586093+0000","last_clean":"2021-02-08T07:14:24.586093+0000","last_became_active":"2021-02-08T07:14:24.585020+0000","last_became_peered":"2021-02-08T07:14:24.585020+0000","last_unstale":"2021-02-08T07:14:24.586093+0000","last_undegraded":"2021-02-08T07:14:24.586093+0000","last_fullsized":"2021-02-08T07:14:24.586093+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-08T07:14:24.587468+0000","last_change":"2021-02-08T07:14:24.586814+0000","last_active":"2021-02-08T07:14:24.587468+0000","last_peered":"2021-02-08T07:14:24.587468+0000","last_clean":"2021-02-08T07:14:24.587468+0000","last_became_active":"2021-02-08T07:14:24.586427+0000","last_became_peered":"2021-02-08T07:14:24.586427+0000","last_unstale":"2021-02-08T07:14:24.587468+0000","last_undegraded":"2021-02-08T07:14:24.587468+0000","last_fullsized":"2021-02-08T07:14:24.587468+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-08T07:14:23.560638+0000","last_clean_scrub_stamp":"2021-02-08T07:14:23.560638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]}]}]}} 2021-02-08T07:14:30.808 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-02-08T07:14:30.808 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-02-08T07:14:30.809 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-02-08T07:14:30.809 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2021-02-08T07:14:31.121 INFO:teuthology.orchestra.run.smithi110.stdout: 2021-02-08T07:14:31.134 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-08T07:14:31.135 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-02-08T07:14:31.135 INFO:teuthology.run_tasks:Running task rgw... 2021-02-08T07:14:31.159 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-02-08T07:14:31.159 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-02-08T07:14:31.159 INFO:tasks.rgw:Creating data pools 2021-02-08T07:14:31.160 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-02-08T07:14:31.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-08T07:14:32.473 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.data' created 2021-02-08T07:14:32.491 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-08T07:14:33.495 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-08T07:14:33.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-08T07:14:34.486 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.index' created 2021-02-08T07:14:34.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-08T07:14:35.522 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-08T07:14:35.538 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-02-08T07:14:35.539 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-08T07:14:35.813 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-08T07:14:35.823 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-08T07:14:36.511 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-08T07:14:36.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-08T07:14:36.797 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-08T07:14:36.808 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-08T07:14:37.527 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-08T07:14:37.540 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-02-08T07:14:37.540 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-08T07:14:38.534 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'default.rgw.buckets.data' created 2021-02-08T07:14:38.549 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-08T07:14:39.548 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-08T07:14:39.566 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-08T07:14:40.564 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'default.rgw.buckets.index' created 2021-02-08T07:14:40.576 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-08T07:14:41.586 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-08T07:14:41.601 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-02-08T07:14:41.602 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-08T07:14:41.867 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-08T07:14:41.877 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-08T07:14:42.567 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-08T07:14:42.578 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-08T07:14:42.850 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-08T07:14:42.860 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-08T07:14:43.576 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-08T07:14:43.587 DEBUG:tasks.rgw:Pools created 2021-02-08T07:14:43.587 INFO:tasks.rgw:Configuring compression type = random 2021-02-08T07:14:43.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-02-08T07:14:43.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-02-08T07:14:43.588 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-02-08T07:14:43.624 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:14:43.624 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:14:43.634 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:43.634+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.554+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.554+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.571 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.570+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.572 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.570+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.572 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.570+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.572 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.570+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.572 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.570+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.594 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.595 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.595 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.595 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.595 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.596 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.596 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 10 cannot find current period zonegroup using local zonegroup 2021-02-08T07:14:45.596 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.596 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.596 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.597 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.597 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 10 Creating default zonegroup 2021-02-08T07:14:45.597 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 20 get_system_obj_state: rctx=0x7fff7dd4a390 obj=:.pools.avail state=0x55e7eb904010 s->prefetch_data=0 2021-02-08T07:14:45.597 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.594+0000 7f84e45e2080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-08T07:14:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.614+0000 7f84e45e2080 10 WARNING: store->list_zones() returned r=-2 2021-02-08T07:14:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.614+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.616 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.614+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.622 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.622+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.622+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.622+0000 7f84e45e2080 10 could not read realm id: (2) No such file or directory 2021-02-08T07:14:45.624 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.622+0000 7f84e45e2080 10 WARNING: failed to set zone as default, r=-22 2021-02-08T07:14:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.622+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.625 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.622+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.630 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.631 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 20 rados_obj.operate() r=0 bl.length=839 2021-02-08T07:14:45.631 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.632 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:14:45.632 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 20 zonegroup default 2021-02-08T07:14:45.632 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.633 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:45.633 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 10 Cannot find current period zone using local zone 2021-02-08T07:14:45.633 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 10 Using default name default 2021-02-08T07:14:45.633 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.630+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.634 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.634+0000 7f84e45e2080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:45.634 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.634+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:45.635 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.634+0000 7f84e45e2080 20 rados_obj.operate() r=0 bl.length=839 2021-02-08T07:14:45.635 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.634+0000 7f84e45e2080 20 zone default found 2021-02-08T07:14:45.635 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:45.634+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:47.596 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:47.594+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:47.596 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:47.594+0000 7f84e45e2080 20 rados->read ofs=0 len=0 2021-02-08T07:14:47.596 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:47.594+0000 7f84e45e2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:47.594+0000 7f84e45e2080 20 started sync module instance, tier type = 2021-02-08T07:14:47.597 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:47.594+0000 7f84e45e2080 20 started zone id=b207a383-491d-42a8-b48c-3a341e8c484f (name=default) with tier type = 2021-02-08T07:14:49.657 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.654+0000 7f84e45e2080 20 add_watcher() i=0 2021-02-08T07:14:49.658 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.654+0000 7f84e45e2080 20 add_watcher() i=1 2021-02-08T07:14:49.658 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.654+0000 7f84e45e2080 20 add_watcher() i=2 2021-02-08T07:14:49.658 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.654+0000 7f84e45e2080 20 add_watcher() i=3 2021-02-08T07:14:49.658 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.654+0000 7f84e45e2080 20 add_watcher() i=4 2021-02-08T07:14:49.658 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.654+0000 7f84e45e2080 20 add_watcher() i=5 2021-02-08T07:14:49.658 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.658+0000 7f84e45e2080 20 add_watcher() i=6 2021-02-08T07:14:49.660 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.658+0000 7f84e45e2080 20 add_watcher() i=7 2021-02-08T07:14:49.660 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.658+0000 7f84e45e2080 2 all 8 watchers are set, enabling cache 2021-02-08T07:14:49.660 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.658+0000 7f84e45e2080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-08T07:14:49.660 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.658+0000 7f84e45e2080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-08T07:14:49.660 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:49.658+0000 7f84a17e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-08T07:14:53.616 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.614+0000 7f84e45e2080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-08T07:14:53.742 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.738+0000 7f84077fe700 20 reqs_thread_entry: start 2021-02-08T07:14:53.742 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.738+0000 7f84057fa700 20 reqs_thread_entry: start 2021-02-08T07:14:53.742 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.738+0000 7f84e45e2080 20 init_complete bucket index max shards: 11 2021-02-08T07:14:53.742 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.738+0000 7f83f37fe700 20 reqs_thread_entry: start 2021-02-08T07:14:53.765 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.762+0000 7f84a17e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-08T07:14:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.766+0000 7f84e45e2080 20 remove_watcher() i=0 2021-02-08T07:14:53.767 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.766+0000 7f84e45e2080 2 removed watcher, disabling cache 2021-02-08T07:14:53.770 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.770+0000 7f84e45e2080 20 remove_watcher() i=1 2021-02-08T07:14:53.776 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.774+0000 7f84e45e2080 20 remove_watcher() i=2 2021-02-08T07:14:53.778 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.778+0000 7f84e45e2080 20 remove_watcher() i=3 2021-02-08T07:14:53.781 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.778+0000 7f84e45e2080 20 remove_watcher() i=4 2021-02-08T07:14:53.784 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.782+0000 7f84e45e2080 20 remove_watcher() i=5 2021-02-08T07:14:53.788 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.786+0000 7f84e45e2080 20 remove_watcher() i=6 2021-02-08T07:14:53.790 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.790+0000 7f84e45e2080 20 remove_watcher() i=7 2021-02-08T07:14:53.799 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:14:53.800 DEBUG:tasks.util.rgw: json result: [] 2021-02-08T07:14:53.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-08T07:14:53.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-08T07:14:53.800 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-08T07:14:53.881 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:14:53.882 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:14:53.892 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.890+0000 7f0f7ecf1080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-08T07:14:53.892 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.890+0000 7f0f7ecf1080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-08T07:14:53.900 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.898+0000 7f0eb27fc700 20 reqs_thread_entry: start 2021-02-08T07:14:53.900 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.898+0000 7f0f7ecf1080 20 rados->read ofs=0 len=0 2021-02-08T07:14:53.902 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.898+0000 7f0f7ecf1080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:53.902 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.898+0000 7f0f7ecf1080 20 rados->read ofs=0 len=0 2021-02-08T07:14:53.903 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.902+0000 7f0f7ecf1080 20 rados_obj.operate() r=0 bl.length=839 2021-02-08T07:14:53.903 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.902+0000 7f0f7ecf1080 20 rados->read ofs=0 len=0 2021-02-08T07:14:53.904 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.902+0000 7f0f7ecf1080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:53.904 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.902+0000 7f0f7ecf1080 20 rados->read ofs=0 len=0 2021-02-08T07:14:53.905 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.902+0000 7f0f7ecf1080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:53.905 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.902+0000 7f0f7ecf1080 20 rados->read ofs=0 len=0 2021-02-08T07:14:53.906 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:53.902+0000 7f0f7ecf1080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:14:53.916 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"b207a383-491d-42a8-b48c-3a341e8c484f","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-02-08T07:14:53.916 DEBUG:tasks.util.rgw: json result: {'id': 'b207a383-491d-42a8-b48c-3a341e8c484f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-02-08T07:14:53.916 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-02-08T07:14:53.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-02-08T07:14:53.917 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-02-08T07:14:53.993 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:14:53.993 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:14:54.005 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.002+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.007 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.006+0000 7f6107982080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.007 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.006+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.008 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.006+0000 7f6107982080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.008 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.006+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.008 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.006+0000 7f6107982080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.009 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.006+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.009 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.006+0000 7f6107982080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.033+0000 7f6107982080 20 RGWRados::pool_iterate: got zone_info.b207a383-491d-42a8-b48c-3a341e8c484f 2021-02-08T07:14:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.033+0000 7f6107982080 20 RGWRados::pool_iterate: got zonegroup_info.35691ba2-72ba-4db2-ae19-52d6a2dabe76 2021-02-08T07:14:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.033+0000 7f6107982080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-08T07:14:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.033+0000 7f6107982080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-08T07:14:54.038 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.033+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.037+0000 7f6107982080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:54.039 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.037+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.037+0000 7f6107982080 20 rados_obj.operate() r=0 bl.length=855 2021-02-08T07:14:54.040 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.037+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.041 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.037+0000 7f6107982080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:54.041 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.037+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.042 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.037+0000 7f6107982080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:14:54.042 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.043 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.043 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 10 cannot find current period zonegroup using local zonegroup 2021-02-08T07:14:54.043 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.044 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.044 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.044 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:54.045 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.046 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:14:54.046 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 zonegroup default 2021-02-08T07:14:54.046 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.041+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.046 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.046 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 10 Cannot find current period zone using local zone 2021-02-08T07:14:54.047 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.047 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.047 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.048 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:54.048 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.049 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 20 rados_obj.operate() r=0 bl.length=855 2021-02-08T07:14:54.049 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 20 zone default found 2021-02-08T07:14:54.049 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.045+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.050 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.049+0000 7f6107982080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.050 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.049+0000 7f6107982080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.051 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.049+0000 7f6107982080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.051 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.049+0000 7f6107982080 20 started sync module instance, tier type = 2021-02-08T07:14:54.051 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.049+0000 7f6107982080 20 started zone id=b207a383-491d-42a8-b48c-3a341e8c484f (name=default) with tier type = 2021-02-08T07:14:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.073+0000 7f6107982080 20 add_watcher() i=0 2021-02-08T07:14:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.073+0000 7f6107982080 20 add_watcher() i=1 2021-02-08T07:14:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.073+0000 7f6107982080 20 add_watcher() i=2 2021-02-08T07:14:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.073+0000 7f6107982080 20 add_watcher() i=3 2021-02-08T07:14:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.073+0000 7f6107982080 20 add_watcher() i=4 2021-02-08T07:14:54.077 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.073+0000 7f6107982080 20 add_watcher() i=5 2021-02-08T07:14:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.077+0000 7f6107982080 20 add_watcher() i=6 2021-02-08T07:14:54.079 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.077+0000 7f6107982080 20 add_watcher() i=7 2021-02-08T07:14:54.080 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.077+0000 7f6107982080 2 all 8 watchers are set, enabling cache 2021-02-08T07:14:54.080 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.077+0000 7f6107982080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-08T07:14:54.080 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.077+0000 7f6107982080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-08T07:14:54.080 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.077+0000 7f60c4fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-08T07:14:54.176 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.173+0000 7f6036ffd700 20 reqs_thread_entry: start 2021-02-08T07:14:54.176 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.173+0000 7f6034ff9700 20 reqs_thread_entry: start 2021-02-08T07:14:54.177 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.173+0000 7f6107982080 20 init_complete bucket index max shards: 11 2021-02-08T07:14:54.177 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.173+0000 7f602affd700 20 reqs_thread_entry: start 2021-02-08T07:14:54.199 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.197+0000 7f60c4fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-08T07:14:54.202 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.197+0000 7f6107982080 20 remove_watcher() i=0 2021-02-08T07:14:54.202 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.197+0000 7f6107982080 2 removed watcher, disabling cache 2021-02-08T07:14:54.205 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.201+0000 7f6107982080 20 remove_watcher() i=1 2021-02-08T07:14:54.207 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.205+0000 7f6107982080 20 remove_watcher() i=2 2021-02-08T07:14:54.210 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.205+0000 7f6107982080 20 remove_watcher() i=3 2021-02-08T07:14:54.213 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.209+0000 7f6107982080 20 remove_watcher() i=4 2021-02-08T07:14:54.216 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.213+0000 7f6107982080 20 remove_watcher() i=5 2021-02-08T07:14:54.219 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.217+0000 7f6107982080 20 remove_watcher() i=6 2021-02-08T07:14:54.222 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.217+0000 7f6107982080 20 remove_watcher() i=7 2021-02-08T07:14:54.231 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:14:54.231 DEBUG:tasks.util.rgw: json result: [] 2021-02-08T07:14:54.231 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-08T07:14:54.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-08T07:14:54.232 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-08T07:14:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:14:54.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:14:54.329 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.325+0000 7fb394171080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-08T07:14:54.330 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.325+0000 7fb394171080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-08T07:14:54.336 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.333+0000 7fb2f8ff9700 20 reqs_thread_entry: start 2021-02-08T07:14:54.336 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.333+0000 7fb394171080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.338 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.333+0000 7fb394171080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:54.338 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.333+0000 7fb394171080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.340 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.337+0000 7fb394171080 20 rados_obj.operate() r=0 bl.length=855 2021-02-08T07:14:54.340 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.337+0000 7fb394171080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.340 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.337+0000 7fb394171080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:54.341 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.337+0000 7fb394171080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.341 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.337+0000 7fb394171080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:54.341 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.337+0000 7fb394171080 20 rados->read ofs=0 len=0 2021-02-08T07:14:54.342 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:14:54.337+0000 7fb394171080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:14:54.353 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"b207a383-491d-42a8-b48c-3a341e8c484f","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-02-08T07:14:54.353 DEBUG:tasks.util.rgw: json result: {'id': 'b207a383-491d-42a8-b48c-3a341e8c484f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-02-08T07:14:54.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-02-08T07:14:54.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-02-08T07:14:54.354 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-02-08T07:14:54.391 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:14:54.391 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:14:54.404 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:54.402+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.526+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.526+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.535 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.534+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.535 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.534+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.535 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.534+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.535 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.534+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.536 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.534+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.564 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.562+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.562+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.562+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.562+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.562+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.562+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.562+0000 7fa88836e080 10 cannot find current period zonegroup using local zonegroup 2021-02-08T07:14:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.562+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.566+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.566+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.566+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.566+0000 7fa88836e080 10 Creating default zonegroup 2021-02-08T07:14:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.566+0000 7fa88836e080 20 get_system_obj_state: rctx=0x7ffff94df8c0 obj=:.pools.avail state=0x55e52ad86010 s->prefetch_data=0 2021-02-08T07:14:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.566+0000 7fa88836e080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-08T07:14:55.580 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.578+0000 7fa88836e080 10 WARNING: store->list_zones() returned r=-2 2021-02-08T07:14:55.580 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.578+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.581 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.578+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.582+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.582+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.582+0000 7fa88836e080 10 could not read realm id: (2) No such file or directory 2021-02-08T07:14:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.582+0000 7fa88836e080 10 WARNING: failed to set zone as default, r=-22 2021-02-08T07:14:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.582+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.582+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.586+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados_obj.operate() r=0 bl.length=839 2021-02-08T07:14:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:14:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 zonegroup default 2021-02-08T07:14:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 10 Cannot find current period zone using local zone 2021-02-08T07:14:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 10 Using default name default 2021-02-08T07:14:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:14:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados_obj.operate() r=0 bl.length=839 2021-02-08T07:14:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 zone default found 2021-02-08T07:14:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:55.590+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:57.562+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:57.562+0000 7fa88836e080 20 rados->read ofs=0 len=0 2021-02-08T07:14:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:57.562+0000 7fa88836e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:14:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:57.562+0000 7fa88836e080 20 started sync module instance, tier type = 2021-02-08T07:14:57.564 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:57.562+0000 7fa88836e080 20 started zone id=28991704-c3d3-41cc-9729-8c0666df8a57 (name=default) with tier type = 2021-02-08T07:14:59.597 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.594+0000 7fa88836e080 20 add_watcher() i=0 2021-02-08T07:14:59.597 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.594+0000 7fa88836e080 20 add_watcher() i=1 2021-02-08T07:14:59.597 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.594+0000 7fa88836e080 20 add_watcher() i=2 2021-02-08T07:14:59.597 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.594+0000 7fa88836e080 20 add_watcher() i=3 2021-02-08T07:14:59.597 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.594+0000 7fa88836e080 20 add_watcher() i=4 2021-02-08T07:14:59.597 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.594+0000 7fa88836e080 20 add_watcher() i=5 2021-02-08T07:14:59.597 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.594+0000 7fa88836e080 20 add_watcher() i=6 2021-02-08T07:14:59.598 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.598+0000 7fa88836e080 20 add_watcher() i=7 2021-02-08T07:14:59.598 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.598+0000 7fa88836e080 2 all 8 watchers are set, enabling cache 2021-02-08T07:14:59.598 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.598+0000 7fa88836e080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-08T07:14:59.598 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.598+0000 7fa88836e080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-08T07:14:59.598 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:14:59.598+0000 7fa8457e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-08T07:15:03.569 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.571+0000 7fa88836e080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-08T07:15:03.674 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.675+0000 7fa7ab7fe700 20 reqs_thread_entry: start 2021-02-08T07:15:03.675 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.675+0000 7fa7a97fa700 20 reqs_thread_entry: start 2021-02-08T07:15:03.675 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.675+0000 7fa88836e080 20 init_complete bucket index max shards: 11 2021-02-08T07:15:03.675 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.675+0000 7fa79b7fe700 20 reqs_thread_entry: start 2021-02-08T07:15:03.700 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.699+0000 7fa8457e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-08T07:15:03.703 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.703+0000 7fa88836e080 20 remove_watcher() i=0 2021-02-08T07:15:03.703 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.703+0000 7fa88836e080 2 removed watcher, disabling cache 2021-02-08T07:15:03.706 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.707+0000 7fa88836e080 20 remove_watcher() i=1 2021-02-08T07:15:03.709 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.711+0000 7fa88836e080 20 remove_watcher() i=2 2021-02-08T07:15:03.711 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.711+0000 7fa88836e080 20 remove_watcher() i=3 2021-02-08T07:15:03.714 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.715+0000 7fa88836e080 20 remove_watcher() i=4 2021-02-08T07:15:03.716 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.715+0000 7fa88836e080 20 remove_watcher() i=5 2021-02-08T07:15:03.719 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.719+0000 7fa88836e080 20 remove_watcher() i=6 2021-02-08T07:15:03.721 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.723+0000 7fa88836e080 20 remove_watcher() i=7 2021-02-08T07:15:03.728 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2021-02-08T07:15:03.729 DEBUG:tasks.util.rgw: json result: [] 2021-02-08T07:15:03.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-08T07:15:03.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-08T07:15:03.729 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-08T07:15:03.811 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:03.811 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:03.823 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.823+0000 7fcaf81ea080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-08T07:15:03.823 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.823+0000 7fcaf81ea080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-08T07:15:03.830 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.831+0000 7fca6cff9700 20 reqs_thread_entry: start 2021-02-08T07:15:03.830 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.831+0000 7fcaf81ea080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.831 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.831+0000 7fcaf81ea080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:15:03.831 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.831+0000 7fcaf81ea080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.832 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.831+0000 7fcaf81ea080 20 rados_obj.operate() r=0 bl.length=839 2021-02-08T07:15:03.832 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.831+0000 7fcaf81ea080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.832 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.835+0000 7fcaf81ea080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.832 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.835+0000 7fcaf81ea080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.833 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.835+0000 7fcaf81ea080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:15:03.833 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.835+0000 7fcaf81ea080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.835 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.835+0000 7fcaf81ea080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:15:03.844 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"28991704-c3d3-41cc-9729-8c0666df8a57","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-02-08T07:15:03.845 DEBUG:tasks.util.rgw: json result: {'id': '28991704-c3d3-41cc-9729-8c0666df8a57', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-02-08T07:15:03.845 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-02-08T07:15:03.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-02-08T07:15:03.845 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-02-08T07:15:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:03.935 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.935+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.937 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.939+0000 7f0968dbb080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.937 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.939+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.938 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.939+0000 7f0968dbb080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.938 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.939+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.939 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.939+0000 7f0968dbb080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.939 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.939+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.939 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.939+0000 7f0968dbb080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:15:03.965 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 RGWRados::pool_iterate: got zonegroup_info.e40ff899-dff5-4793-b323-b5c25baa1d6a 2021-02-08T07:15:03.965 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 RGWRados::pool_iterate: got zone_info.28991704-c3d3-41cc-9729-8c0666df8a57 2021-02-08T07:15:03.966 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-08T07:15:03.966 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-08T07:15:03.966 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.966 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:15:03.966 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.967 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 rados_obj.operate() r=0 bl.length=855 2021-02-08T07:15:03.967 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.968 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:15:03.969 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.967+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.969 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.971+0000 7f0968dbb080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:15:03.969 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.971+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.970 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.971+0000 7f0968dbb080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.970 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.971+0000 7f0968dbb080 10 cannot find current period zonegroup using local zonegroup 2021-02-08T07:15:03.970 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.971+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.971 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.971+0000 7f0968dbb080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.971 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.971+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.972 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.971+0000 7f0968dbb080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:15:03.972 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.971+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.973 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:15:03.973 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 zonegroup default 2021-02-08T07:15:03.973 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.973 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.974 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 10 Cannot find current period zone using local zone 2021-02-08T07:15:03.974 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.974 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.974 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.975 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:15:03.975 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.976 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados_obj.operate() r=0 bl.length=855 2021-02-08T07:15:03.976 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 zone default found 2021-02-08T07:15:03.976 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.975+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.977 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.979+0000 7f0968dbb080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.977 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.979+0000 7f0968dbb080 20 rados->read ofs=0 len=0 2021-02-08T07:15:03.978 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.979+0000 7f0968dbb080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:03.978 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.979+0000 7f0968dbb080 20 started sync module instance, tier type = 2021-02-08T07:15:03.978 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:03.979+0000 7f0968dbb080 20 started zone id=28991704-c3d3-41cc-9729-8c0666df8a57 (name=default) with tier type = 2021-02-08T07:15:04.001 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 add_watcher() i=0 2021-02-08T07:15:04.001 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 add_watcher() i=1 2021-02-08T07:15:04.001 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 add_watcher() i=2 2021-02-08T07:15:04.001 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 add_watcher() i=3 2021-02-08T07:15:04.001 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 add_watcher() i=4 2021-02-08T07:15:04.001 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 add_watcher() i=5 2021-02-08T07:15:04.001 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 add_watcher() i=6 2021-02-08T07:15:04.002 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 add_watcher() i=7 2021-02-08T07:15:04.002 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 2 all 8 watchers are set, enabling cache 2021-02-08T07:15:04.007 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-08T07:15:04.007 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f0968dbb080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-08T07:15:04.007 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.003+0000 7f09257e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-08T07:15:04.087 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.087+0000 7f08977fe700 20 reqs_thread_entry: start 2021-02-08T07:15:04.087 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.087+0000 7f08957fa700 20 reqs_thread_entry: start 2021-02-08T07:15:04.088 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.087+0000 7f0968dbb080 20 init_complete bucket index max shards: 11 2021-02-08T07:15:04.088 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.087+0000 7f08837fe700 20 reqs_thread_entry: start 2021-02-08T07:15:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.115+0000 7f09257e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-08T07:15:04.116 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.115+0000 7f0968dbb080 20 remove_watcher() i=0 2021-02-08T07:15:04.116 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.115+0000 7f0968dbb080 2 removed watcher, disabling cache 2021-02-08T07:15:04.119 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.119+0000 7f0968dbb080 20 remove_watcher() i=1 2021-02-08T07:15:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.123+0000 7f0968dbb080 20 remove_watcher() i=2 2021-02-08T07:15:04.126 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.127+0000 7f0968dbb080 20 remove_watcher() i=3 2021-02-08T07:15:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.127+0000 7f0968dbb080 20 remove_watcher() i=4 2021-02-08T07:15:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.131+0000 7f0968dbb080 20 remove_watcher() i=5 2021-02-08T07:15:04.134 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.135+0000 7f0968dbb080 20 remove_watcher() i=6 2021-02-08T07:15:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.139+0000 7f0968dbb080 20 remove_watcher() i=7 2021-02-08T07:15:04.144 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2021-02-08T07:15:04.144 DEBUG:tasks.util.rgw: json result: [] 2021-02-08T07:15:04.144 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-08T07:15:04.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-08T07:15:04.144 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-08T07:15:04.225 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.235+0000 7f3c8cee8080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-08T07:15:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.235+0000 7f3c8cee8080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-08T07:15:04.242 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.243+0000 7f3bc97fa700 20 reqs_thread_entry: start 2021-02-08T07:15:04.243 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.243+0000 7f3c8cee8080 20 rados->read ofs=0 len=0 2021-02-08T07:15:04.245 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.247+0000 7f3c8cee8080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:15:04.245 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.247+0000 7f3c8cee8080 20 rados->read ofs=0 len=0 2021-02-08T07:15:04.246 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.247+0000 7f3c8cee8080 20 rados_obj.operate() r=0 bl.length=855 2021-02-08T07:15:04.246 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.247+0000 7f3c8cee8080 20 rados->read ofs=0 len=0 2021-02-08T07:15:04.246 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.247+0000 7f3c8cee8080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-08T07:15:04.246 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.247+0000 7f3c8cee8080 20 rados->read ofs=0 len=0 2021-02-08T07:15:04.247 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.247+0000 7f3c8cee8080 20 rados_obj.operate() r=0 bl.length=46 2021-02-08T07:15:04.247 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.247+0000 7f3c8cee8080 20 rados->read ofs=0 len=0 2021-02-08T07:15:04.248 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:04.247+0000 7f3c8cee8080 20 rados_obj.operate() r=0 bl.length=358 2021-02-08T07:15:04.256 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"28991704-c3d3-41cc-9729-8c0666df8a57","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-02-08T07:15:04.257 DEBUG:tasks.util.rgw: json result: {'id': '28991704-c3d3-41cc-9729-8c0666df8a57', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-02-08T07:15:04.257 INFO:tasks.rgw:Starting rgw... 2021-02-08T07:15:04.257 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-02-08T07:15:04.257 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-02-08T07:15:04.257 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-08T07:15:04.258 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-08T07:15:04.260 INFO:tasks.rgw.c1.client.0:Started 2021-02-08T07:15:04.260 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-02-08T07:15:04.260 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-02-08T07:15:04.260 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-08T07:15:04.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-08T07:15:04.262 INFO:tasks.rgw.c1.client.1:Started 2021-02-08T07:15:04.263 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-02-08T07:15:04.263 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-02-08T07:15:04.263 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-08T07:15:04.263 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-08T07:15:04.302 INFO:tasks.rgw.c2.client.0:Started 2021-02-08T07:15:04.302 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-02-08T07:15:04.302 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-02-08T07:15:04.303 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-08T07:15:04.303 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-08T07:15:04.305 INFO:tasks.rgw.c2.client.1:Started 2021-02-08T07:15:04.305 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi017.front.sepia.ceph.com:8000/ 2021-02-08T07:15:04.305 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000/ 2021-02-08T07:15:04.329 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:04.330 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:04.334 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8000: Connection refused 2021-02-08T07:15:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-08T07:15:05.335 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000/ 2021-02-08T07:15:05.349 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:05.349 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:05.353 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2021-02-08T07:15:05.354 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2021-02-08T07:15:05.354 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi017.front.sepia.ceph.com:8001/ 2021-02-08T07:15:05.354 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001/ 2021-02-08T07:15:05.409 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:05.409 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:05.414 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-02-08T07:15:05.414 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2021-02-08T07:15:05.415 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi110.front.sepia.ceph.com:8000/ 2021-02-08T07:15:05.415 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:8000/ 2021-02-08T07:15:05.447 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:05.447 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:05.453 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 19454 0 --:--:-- --:--:-- --:--:-- 42800 2021-02-08T07:15:05.453 INFO:teuthology.orchestra.run.smithi110.stdout:anonymous 2021-02-08T07:15:05.454 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi110.front.sepia.ceph.com:8001/ 2021-02-08T07:15:05.454 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:8001/ 2021-02-08T07:15:05.511 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:05.511 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:05.516 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2021-02-08T07:15:05.517 INFO:teuthology.orchestra.run.smithi110.stdout:anonymous 2021-02-08T07:15:05.518 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-02-08T07:15:06.016 INFO:tasks.rgw_multisite:creating realm.. 2021-02-08T07:15:06.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:06.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:06.016 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:06.055 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:06.055 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:06.113 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"c186a644-2499-4387-8680-a2142bd4cec7","name":"test-realm","current_period":"6c807ecd-9403-4d96-a6e0-d3fae5d082e8","epoch":1} 2021-02-08T07:15:06.113 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-02-08T07:15:06.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:06.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:06.114 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:06.194 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:06.194 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:06.218 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:15:06.213+0000 7f69e81b5080 -1 old format 2021-02-08T07:15:06.235 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}} 2021-02-08T07:15:06.235 INFO:tasks.rgw_multisite:creating master zone.. 2021-02-08T07:15:06.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:06.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:06.236 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key SOBS5D2YHN3GGT4Z --secret 1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG --endpoints http://smithi017.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:06.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:06.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:06.415 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:15:06.416 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-02-08T07:15:07.659 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-02-08T07:15:07.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-02-08T07:15:08.691 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-02-08T07:15:08.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:08.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:08.708 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:08.746 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:08.747 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:08.774 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-02-08T07:15:08.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:08.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:08.775 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:08.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:08.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:16.936 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"d4b11cd5-1a7c-4383-afcf-402844a1791d","epoch":1,"predecessor_uuid":"6c807ecd-9403-4d96-a6e0-d3fae5d082e8","sync_status":[],"period_map":{"id":"d4b11cd5-1a7c-4383-afcf-402844a1791d","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":2} 2021-02-08T07:15:16.936 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-02-08T07:15:16.936 INFO:teuthology.orchestra.run:waiting for 300 2021-02-08T07:15:16.980 INFO:tasks.rgw.c1.client.0.smithi017.stdout:2021-02-08T07:15:16.977+0000 7fa4b16c0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 14052) UID: 0 2021-02-08T07:15:16.980 INFO:tasks.rgw.c1.client.0.smithi017.stdout:2021-02-08T07:15:16.977+0000 7fa4c8d820c0 -1 shutting down 2021-02-08T07:15:21.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-02-08T07:15:23.037 INFO:tasks.rgw.c1.client.0:Stopped 2021-02-08T07:15:23.038 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-08T07:15:23.038 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-08T07:15:23.040 INFO:tasks.rgw.c1.client.0:Started 2021-02-08T07:15:23.041 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000 2021-02-08T07:15:23.057 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:23.058 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:23.061 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8000: Connection refused 2021-02-08T07:15:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-08T07:15:24.063 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000 2021-02-08T07:15:24.078 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:24.078 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:24.082 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 53500 2021-02-08T07:15:24.083 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2021-02-08T07:15:24.083 INFO:tasks.rgw_multisite:creating admin user.. 2021-02-08T07:15:24.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:24.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:24.084 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key SOBS5D2YHN3GGT4Z --secret 1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:24.177 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:24.177 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:24.359 INFO:teuthology.orchestra.run.smithi017.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-08T07:15:24.388 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-08T07:15:24.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:24.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:24.389 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi017.front.sepia.ceph.com:8000 --access-key SOBS5D2YHN3GGT4Z --secret 1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:24.431 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:24.432 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:24.545 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"c186a644-2499-4387-8680-a2142bd4cec7","name":"test-realm","current_period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","epoch":2} 2021-02-08T07:15:24.545 INFO:tasks.rgw_multisite:creating zone.. 2021-02-08T07:15:24.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--endpoints', 'http://smithi110.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:24.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--endpoints', 'http://smithi110.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:24.546 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key SOBS5D2YHN3GGT4Z --secret 1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG --endpoints http://smithi110.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:24.631 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:24.632 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:24.694 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:24.695+0000 7fb5cde75080 0 failed reading obj info from .rgw.root:zone_info.757db88c-ee56-405d-bd0c-79b9f5b05722: (2) No such file or directory 2021-02-08T07:15:24.694 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:15:24.695+0000 7fb5cde75080 0 WARNING: could not read zone params for zone id=757db88c-ee56-405d-bd0c-79b9f5b05722 name=test-zone1 2021-02-08T07:15:24.709 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:15:24.710 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-02-08T07:15:25.551 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-02-08T07:15:25.568 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-02-08T07:15:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-02-08T07:15:26.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:26.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:26.600 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:26.634 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:26.635 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:26.658 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-02-08T07:15:26.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:26.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:26.659 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:26.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-02-08T07:15:34.741 INFO:teuthology.orchestra.run.smithi110.stderr:Sending period to new master zone 757db88c-ee56-405d-bd0c-79b9f5b05722 2021-02-08T07:15:35.838 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"d4b11cd5-1a7c-4383-afcf-402844a1791d","epoch":2,"predecessor_uuid":"6c807ecd-9403-4d96-a6e0-d3fae5d082e8","sync_status":[],"period_map":{"id":"d4b11cd5-1a7c-4383-afcf-402844a1791d","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":2} 2021-02-08T07:15:35.839 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-02-08T07:15:35.839 INFO:teuthology.orchestra.run:waiting for 300 2021-02-08T07:15:35.885 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:15:35.887+0000 7f05bffff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 14059) UID: 0 2021-02-08T07:15:35.885 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:15:35.887+0000 7f05e3c9f0c0 -1 shutting down 2021-02-08T07:15:37.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-02-08T07:15:41.941 INFO:tasks.rgw.c2.client.0:Stopped 2021-02-08T07:15:41.942 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-08T07:15:41.942 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-08T07:15:41.946 INFO:tasks.rgw.c2.client.0:Started 2021-02-08T07:15:41.946 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:8000 2021-02-08T07:15:41.964 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:41.964 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:41.968 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi110.front.sepia.ceph.com port 8000: Connection refused 2021-02-08T07:15:41.969 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-08T07:15:42.969 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:8000 2021-02-08T07:15:42.988 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:42.988 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:42.993 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-02-08T07:15:42.993 INFO:teuthology.orchestra.run.smithi110.stdout:anonymous 2021-02-08T07:15:42.994 INFO:tasks.rgw_multisite:creating zone.. 2021-02-08T07:15:42.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:42.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:42.994 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key SOBS5D2YHN3GGT4Z --secret 1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG --endpoints http://smithi017.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:43.037 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:43.038 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:43.106 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:15:43.101+0000 7f58cc52d080 0 failed reading obj info from .rgw.root:zone_info.c897876e-5513-44a9-a99f-ee5993968e59: (2) No such file or directory 2021-02-08T07:15:43.106 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:15:43.101+0000 7f58cc52d080 0 WARNING: could not read zone params for zone id=c897876e-5513-44a9-a99f-ee5993968e59 name=test-zone2 2021-02-08T07:15:43.116 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:15:43.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-02-08T07:15:43.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-02-08T07:15:43.510 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-02-08T07:15:43.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-02-08T07:15:44.544 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-02-08T07:15:44.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:44.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:44.557 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:44.626 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:44.627 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:44.650 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-02-08T07:15:44.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:44.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:15:44.652 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:15:44.687 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:15:44.687 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:15:52.743 INFO:teuthology.orchestra.run.smithi017.stderr:Sending period to new master zone 757db88c-ee56-405d-bd0c-79b9f5b05722 2021-02-08T07:15:53.847 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"d4b11cd5-1a7c-4383-afcf-402844a1791d","epoch":3,"predecessor_uuid":"6c807ecd-9403-4d96-a6e0-d3fae5d082e8","sync_status":[],"period_map":{"id":"d4b11cd5-1a7c-4383-afcf-402844a1791d","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":2} 2021-02-08T07:15:53.848 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-08T07:15:53.848 INFO:teuthology.orchestra.run:waiting for 300 2021-02-08T07:15:53.891 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:15:53.889+0000 7f23a67fc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 14055) UID: 0 2021-02-08T07:15:53.891 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:15:53.889+0000 7f23ca0560c0 -1 shutting down 2021-02-08T07:15:55.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-08T07:15:59.950 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-08T07:15:59.950 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-08T07:15:59.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-08T07:15:59.953 INFO:tasks.rgw.c1.client.1:Started 2021-02-08T07:15:59.953 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2021-02-08T07:15:59.970 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:15:59.970 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:15:59.974 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2021-02-08T07:15:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-08T07:16:00.976 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2021-02-08T07:16:00.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-08T07:16:00.993 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:16:00.993 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:16:00.994 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-02-08T07:16:00.994 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2021-02-08T07:16:00.994 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-08T07:16:00.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:00.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:00.995 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi017.front.sepia.ceph.com:8000 --access-key SOBS5D2YHN3GGT4Z --secret 1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG --url http://smithi017.front.sepia.ceph.com:8000 --access-key SOBS5D2YHN3GGT4Z --secret 1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:16:01.041 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:01.042 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:01.086 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:16:01.087+0000 7f8485e03700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.c186a644-2499-4387-8680-a2142bd4cec7.control err=-107 2021-02-08T07:16:01.110 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"c186a644-2499-4387-8680-a2142bd4cec7","name":"test-realm","current_period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","epoch":2} 2021-02-08T07:16:01.111 INFO:tasks.rgw_multisite:creating zone.. 2021-02-08T07:16:01.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--endpoints', 'http://smithi110.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:01.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--endpoints', 'http://smithi110.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:01.111 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key SOBS5D2YHN3GGT4Z --secret 1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG --endpoints http://smithi110.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:16:01.160 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:01.160 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:01.227 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:16:01.227+0000 7f7a21d60080 0 failed reading obj info from .rgw.root:zone_info.757db88c-ee56-405d-bd0c-79b9f5b05722: (2) No such file or directory 2021-02-08T07:16:01.227 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:16:01.227+0000 7f7a21d60080 0 WARNING: could not read zone params for zone id=757db88c-ee56-405d-bd0c-79b9f5b05722 name=test-zone1 2021-02-08T07:16:01.227 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:16:01.227+0000 7f7a21d60080 0 failed reading obj info from .rgw.root:zone_info.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d: (2) No such file or directory 2021-02-08T07:16:01.228 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:16:01.231+0000 7f7a21d60080 0 WARNING: could not read zone params for zone id=8e9c5146-39d0-40f8-b41d-7c27e66f6e3d name=test-zone3 2021-02-08T07:16:01.238 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:16:01.238 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-02-08T07:16:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-02-08T07:16:01.573 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-02-08T07:16:02.576 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-02-08T07:16:02.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:02.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:02.594 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:16:02.647 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:02.647 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:02.671 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":""} 2021-02-08T07:16:02.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:02.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:02.671 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:16:02.706 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:02.706 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:Sending period to new master zone 757db88c-ee56-405d-bd0c-79b9f5b05722 2021-02-08T07:16:12.040 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"d4b11cd5-1a7c-4383-afcf-402844a1791d","epoch":4,"predecessor_uuid":"6c807ecd-9403-4d96-a6e0-d3fae5d082e8","sync_status":[],"period_map":{"id":"d4b11cd5-1a7c-4383-afcf-402844a1791d","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":2} 2021-02-08T07:16:12.040 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-02-08T07:16:12.041 INFO:teuthology.orchestra.run:waiting for 300 2021-02-08T07:16:12.084 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:16:12.087+0000 7f12848d1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 14061) UID: 0 2021-02-08T07:16:12.085 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:16:12.087+0000 7f129bf930c0 -1 shutting down 2021-02-08T07:16:17.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-02-08T07:16:18.142 INFO:tasks.rgw.c2.client.1:Stopped 2021-02-08T07:16:18.142 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-08T07:16:18.142 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-08T07:16:18.145 INFO:tasks.rgw.c2.client.1:Started 2021-02-08T07:16:18.145 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:8001 2021-02-08T07:16:18.163 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:16:18.163 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:16:18.167 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi110.front.sepia.ceph.com port 8001: Connection refused 2021-02-08T07:16:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-08T07:16:19.169 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:8001 2021-02-08T07:16:19.184 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:16:19.184 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:16:19.189 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 26800 0 --:--:-- --:--:-- --:--:-- 33500 2021-02-08T07:16:19.190 INFO:teuthology.orchestra.run.smithi110.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-006020e543-10e6-test-zone4","HostId":"10e6-test-zone4-test-zonegroup"} 2021-02-08T07:16:19.190 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-02-08T07:16:19.190 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-02-08T07:16:19.270 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-02-08T07:16:19.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:19.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:16:19.271 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:16:19.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:19.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:19.541 INFO:teuthology.orchestra.run.smithi017.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"RXDS984K7IRT1PRB04SW","secret_key":"xAQDsTWlyJHmI4gte8HOciKt6rgMJ6joR9WAVCZt"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-08T07:16:19.573 INFO:rgw_multi.tests:meta checkpoint 2021-02-08T07:16:19.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:19.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:19.574 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:16:19.611 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:19.611 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:19.821 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:16:19.858 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-08T07:16:19.858 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:16:19.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:19.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:19.859 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:16:19.902 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:19.902 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:20.108 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:16:20.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:16:23.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-02-08T07:16:25.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:25.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:25.479 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:16:25.519 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:25.520 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:25.728 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:16:25.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:16:30.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:30.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:30.730 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:16:30.770 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:30.770 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:30.976 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:16:30.977 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:16:35.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:35.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:35.978 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:16:36.018 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:36.018 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:36.231 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:16:36.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:16:41.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:41.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:41.232 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:16:41.273 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:41.274 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:41.482 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:16:41.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:16:46.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:46.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:46.484 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:16:46.524 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:46.524 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:46.721 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:16:46.721 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:16:51.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:51.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:51.723 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:16:51.765 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:51.765 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:51.970 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:16:51.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:16:56.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:56.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:16:56.972 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:16:57.016 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:16:57.016 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:16:57.228 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:16:57.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:02.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:02.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:02.231 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:02.271 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:02.271 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:02.483 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:02.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:07.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:07.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:07.485 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:07.524 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:07.524 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:07.728 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:07.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:12.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:12.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:12.730 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:12.976 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:12.977 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:17.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:17.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:17.979 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:18.020 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:18.020 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:18.231 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:18.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:23.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:23.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:23.233 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:23.470 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:23.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:28.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:28.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:28.472 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:28.513 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:28.513 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:28.701 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:28.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:33.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:33.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:33.703 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:33.743 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:33.743 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:33.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:38.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:38.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:38.951 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:38.987 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:38.987 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:39.205 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:39.205 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:44.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:44.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:44.207 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:44.249 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:44.249 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:44.455 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:44.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:49.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:49.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:49.458 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:49.498 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:49.498 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:49.710 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:49.710 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:54.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:54.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:54.712 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:17:54.752 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:17:54.752 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:17:54.962 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:17:54.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:17:59.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:59.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:17:59.964 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:00.005 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:00.005 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:00.226 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:00.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:18:05.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:05.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:05.228 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:05.268 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:05.268 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:05.474 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:05.475 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:18:10.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:10.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:10.476 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:10.517 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:10.517 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:10.714 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:10.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1612768579.532632_32.1 target= 2021-02-08T07:18:15.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:15.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:15.715 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:15.755 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:15.755 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:15.970 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:15.971 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:18:15.971 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:18:15.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:15.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:15.971 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:16.012 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:16.013 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:16.254 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:16.255 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:18:16.255 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:18:16.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:16.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:16.255 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:16.294 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:16.295 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:16.567 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:16.568 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:18:16.568 INFO:rgw_multi.tests:meta checkpoint 2021-02-08T07:18:16.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:16.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:16.568 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:16.609 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:16.609 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:16.809 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:18:16.847 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-08T07:18:16.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:18:16.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:16.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:16.848 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:16.890 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:16.891 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:17.103 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:17.104 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:18:17.104 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:18:17.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:17.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:17.104 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:17.146 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:17.146 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:17.376 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:17.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:18:17.376 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:18:17.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:17.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:17.376 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:17.682 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:17.682 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:18:17.683 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2021-02-08T07:18:17.689 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-1 2021-02-08T07:18:21.683 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-2 2021-02-08T07:18:24.855 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-3 2021-02-08T07:18:24.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:24.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:24.914 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:24.947 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:24.947 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:25.145 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:18:25.181 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768704.893108_51.1","last_update":"2021-02-08T07:18:24.893108Z"},{"marker":"","last_update":"0.000000"}] 2021-02-08T07:18:25.182 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:18:25.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:25.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:25.182 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:25.222 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:25.222 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:25.437 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768701.714859_47.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:21.714859Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:25.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1612768704.893108_51.1 target=1_1612768701.714859_47.1 2021-02-08T07:18:30.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:30.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:30.439 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:30.479 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:30.479 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:30.680 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768704.893108_51.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:24.893108Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:30.681 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:18:30.681 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:18:30.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:30.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:30.681 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:30.722 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:30.722 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:30.961 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768704.893108_51.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:24.893108Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:30.962 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:18:30.962 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:18:30.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:30.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:30.963 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:31.004 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:31.004 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:31.274 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768704.893108_51.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:24.893108Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:31.274 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:18:32.763 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2021-02-08T07:18:32.763 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-4 2021-02-08T07:18:32.794 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-5 2021-02-08T07:18:32.853 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-6 2021-02-08T07:18:32.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:32.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:32.916 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:32.951 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:33.151 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:18:33.184 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768712.891808_63.1","last_update":"2021-02-08T07:18:32.891808Z"},{"marker":"","last_update":"0.000000"}] 2021-02-08T07:18:33.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:18:33.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:33.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:33.185 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:33.433 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768704.893108_51.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:24.893108Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:33.433 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1612768712.891808_63.1 target=1_1612768704.893108_51.1 2021-02-08T07:18:38.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:38.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:38.434 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:38.475 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:38.475 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:38.668 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768712.891808_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:32.891808Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:38.668 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:18:38.668 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:18:38.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:38.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:38.669 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:38.713 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:38.713 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:38.939 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768712.891808_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:32.891808Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:38.940 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:18:38.940 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:18:38.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:38.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:38.940 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:38.982 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:39.250 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768712.891808_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:32.891808Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:39.250 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:18:41.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:41.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:41.193 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:41.226 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:41.227 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:41.454 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:18:41.488 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768721.063275_99.1","last_update":"2021-02-08T07:18:41.063275Z"},{"marker":"","last_update":"0.000000"}] 2021-02-08T07:18:41.488 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:18:41.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:41.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:41.489 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:41.531 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:41.531 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:41.716 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768720.549664_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:40.549664Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:41.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1612768721.063275_99.1 target=1_1612768720.549664_73.1 2021-02-08T07:18:46.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:46.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:46.718 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:46.760 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:46.760 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:46.972 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768721.063275_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:41.063275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:46.972 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:18:46.973 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:18:46.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:46.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:46.973 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:47.014 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:47.014 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:47.258 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768721.063275_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:41.063275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:47.259 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:18:47.259 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:18:47.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:47.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:47.259 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:47.301 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:47.301 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:47.571 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768721.063275_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:41.063275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:47.572 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:18:49.061 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2021-02-08T07:18:49.061 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-7 2021-02-08T07:18:49.362 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-8 2021-02-08T07:18:49.424 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-9 2021-02-08T07:18:49.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:49.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:49.486 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:49.524 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:49.718 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:18:49.756 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768729.462500_111.1","last_update":"2021-02-08T07:18:49.462500Z"},{"marker":"","last_update":"0.000000"}] 2021-02-08T07:18:49.757 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:18:49.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:49.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:49.757 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:50.012 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768721.063275_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:41.063275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:50.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1612768729.462500_111.1 target=1_1612768721.063275_99.1 2021-02-08T07:18:55.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:55.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:55.013 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:55.053 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:55.267 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768729.462500_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:49.462500Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:55.268 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:18:55.268 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:18:55.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:55.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:55.268 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:55.309 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:55.309 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:55.577 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768729.462500_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:49.462500Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:55.578 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:18:55.578 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:18:55.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:55.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:55.578 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:55.615 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:55.615 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:18:55.883 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768729.462500_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:49.462500Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:18:55.883 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:18:59.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:59.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:18:59.783 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:18:59.819 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:18:59.819 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:00.030 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:19:00.067 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"","last_update":"0.000000"}] 2021-02-08T07:19:00.067 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:19:00.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:00.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:00.068 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:00.325 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768738.742706_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:58.742706Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:00.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1612768739.752484_117.1 target=1_1612768738.742706_115.1 2021-02-08T07:19:05.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:05.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:05.326 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:05.585 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:05.586 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:19:05.586 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:19:05.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:05.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:05.587 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:05.633 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:05.633 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:05.866 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:05.867 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:19:05.867 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:19:05.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:05.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:05.867 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:05.907 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:05.908 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:06.181 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:06.182 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:19:06.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2021-02-08T07:19:06.935 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-10 2021-02-08T07:19:06.969 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-11 2021-02-08T07:19:07.022 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-12 2021-02-08T07:19:07.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:07.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:07.707 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:07.743 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:07.743 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:07.948 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:19:07.984 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612768747.057366_66.1","last_update":"2021-02-08T07:19:07.057366Z"}] 2021-02-08T07:19:07.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:19:07.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:07.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:07.985 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:08.034 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:08.034 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:08.229 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768747.057366_66.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:19:07.057366Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:08.229 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:19:08.229 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:19:08.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:08.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:08.230 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:08.274 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:08.275 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:08.477 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768747.057366_66.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:19:07.057366Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:08.477 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:19:08.477 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:19:08.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:08.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:08.478 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:08.517 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:08.518 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:08.777 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768747.057366_66.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:19:07.057366Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:08.778 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:19:08.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:08.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:08.778 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:08.817 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:08.818 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:09.059 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.13.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.14.5"},{"key":8,"val":"00000000001.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:09.064 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-10 2021-02-08T07:19:09.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:09.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:09.070 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:09.128 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:09.336 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.19.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5"}}},{"key":8,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.18.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:09.336 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-10 2021-02-08T07:19:09.337 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-10 zones={test-zone2, test-zone1} 2021-02-08T07:19:09.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:09.600 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:09.600 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:09.600 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:09.601 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:09.601 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:09.611 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:09.611 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:09.611 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:09.612 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:09.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:09.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:09.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:09.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:09.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:10.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:10.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:10.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:10.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:10.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:10.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:10.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:10.724 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-10 zones={test-zone2, test-zone1} 2021-02-08T07:19:10.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:10.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:10.724 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:10.760 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:10.760 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:11.003 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.13.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.14.5"},{"key":8,"val":"00000000001.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:11.003 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-10 2021-02-08T07:19:11.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:11.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:11.004 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:11.089 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:11.089 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:11.358 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.19.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:11.359 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-10 2021-02-08T07:19:11.359 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-10 zones={test-zone3, test-zone1} 2021-02-08T07:19:11.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:11.398 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:11.398 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:11.398 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:11.398 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:11.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:11.406 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:11.406 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:11.406 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:11.406 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:11.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:11.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:11.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:11.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:11.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:11.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:11.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:12.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:12.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:12.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:12.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:12.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:12.532 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-10 zones={test-zone3, test-zone1} 2021-02-08T07:19:12.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:12.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:12.532 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:12.573 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:12.574 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:12.775 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.13.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.14.5"},{"key":8,"val":"00000000001.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:12.775 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-10 2021-02-08T07:19:12.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:12.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:12.776 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:12.821 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:13.085 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.19.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.14.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.18.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:13.085 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-10 2021-02-08T07:19:13.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:13.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:13.086 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:13.168 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:13.169 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:13.374 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.4.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.10.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.5.5"},{"key":8,"val":"00000000001.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:13.374 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-11 2021-02-08T07:19:13.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:13.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:13.375 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:13.421 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:13.677 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.4.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:13.677 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-11 2021-02-08T07:19:13.677 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-11 zones={test-zone1, test-zone2} 2021-02-08T07:19:14.672 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:14.681 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:14.681 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:14.681 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:14.681 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:14.682 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:14.688 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:14.688 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:14.688 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:14.688 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:14.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:14.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:14.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:15.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:15.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:15.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:15.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:15.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:15.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:15.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:15.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:15.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:16.033 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-11 zones={test-zone1, test-zone2} 2021-02-08T07:19:16.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:16.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:16.033 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:16.078 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:16.078 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:16.291 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.4.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.10.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.5.5"},{"key":8,"val":"00000000001.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:16.291 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-11 2021-02-08T07:19:16.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:16.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:16.292 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:16.349 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:16.349 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:16.607 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.4.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:16.608 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-11 2021-02-08T07:19:16.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-11 zones={test-zone3, test-zone2} 2021-02-08T07:19:18.247 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:18.256 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:18.257 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:18.263 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:18.263 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:18.263 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:18.263 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:18.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:18.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:18.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:18.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:18.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:18.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:18.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:18.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:19.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:19.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:19.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:19.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:19.441 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-11 zones={test-zone3, test-zone2} 2021-02-08T07:19:19.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:19.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:19.442 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:19.483 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:19.484 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:19.724 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.10.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.3.5"},{"key":8,"val":"00000000001.7.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:19.725 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-12 2021-02-08T07:19:19.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:19.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:19.725 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:19.809 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:19.809 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:20.080 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.3.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:20.081 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-12 2021-02-08T07:19:20.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-12 zones={test-zone1, test-zone3} 2021-02-08T07:19:20.254 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:20.259 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:20.259 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:20.259 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:20.259 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:20.259 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:20.264 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:20.264 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:20.264 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:20.264 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:20.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:20.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:20.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:20.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:20.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:20.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:20.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:20.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:21.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:21.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:21.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:21.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:21.424 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-12 zones={test-zone1, test-zone3} 2021-02-08T07:19:21.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:21.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:21.424 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:21.463 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:21.464 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:21.682 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.10.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.3.5"},{"key":8,"val":"00000000001.7.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:21.682 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-12 2021-02-08T07:19:21.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:21.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:21.683 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:21.726 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:21.726 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:21.961 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.3.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:21.961 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-12 2021-02-08T07:19:21.961 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-12 zones={test-zone2, test-zone3} 2021-02-08T07:19:22.035 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:22.041 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:22.041 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:22.041 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:22.041 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:22.041 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:22.048 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-08T07:19:22.048 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-08T07:19:22.048 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-08T07:19:22.049 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:22.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:22.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:22.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-08T07:19:22.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:22.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:22.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-08T07:19:22.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:22.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:22.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-08T07:19:22.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:22.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:23.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:23.211 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-12 zones={test-zone2, test-zone3} 2021-02-08T07:19:23.211 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2021-02-08T07:19:23.212 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-13 2021-02-08T07:19:23.474 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-14 2021-02-08T07:19:23.537 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-15 2021-02-08T07:19:23.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:23.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:23.743 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:24.000 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:19:24.037 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612768763.570467_145.1","last_update":"2021-02-08T07:19:23.570467Z"}] 2021-02-08T07:19:24.038 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:19:24.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:24.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:24.038 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:24.078 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:24.078 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:24.182 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:19:24.187+0000 7f868a7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:19:24.274 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768747.057366_66.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:19:07.057366Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:24.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612768763.570467_145.1 target=1_1612768747.057366_66.1 2021-02-08T07:19:29.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:29.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:29.276 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:29.530 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768763.570467_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:19:23.570467Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:29.531 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:19:29.531 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:19:29.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:29.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:29.532 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:29.575 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:29.575 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:29.807 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768763.570467_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:19:23.570467Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:29.807 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:19:29.807 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:19:29.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:29.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:29.808 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:29.850 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:29.850 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:30.124 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768763.570467_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:19:23.570467Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:19:30.125 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:19:30.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:30.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:30.125 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:30.168 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:30.168 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:30.404 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.22.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:30.405 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:30.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:30.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:30.405 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:30.664 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.22.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:30.664 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:30.665 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-13 zones={test-zone2, test-zone1} 2021-02-08T07:19:32.306 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:32.314 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:32.314 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:32.367 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:32.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:32.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:32.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:32.712 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-13 zones={test-zone2, test-zone1} 2021-02-08T07:19:32.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:32.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:32.713 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:32.750 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:32.750 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:32.976 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.22.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:32.977 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:32.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:32.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:32.977 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:33.062 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:33.063 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:33.325 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.22.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:33.326 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:33.326 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-13 zones={test-zone3, test-zone1} 2021-02-08T07:19:33.606 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:33.655 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:33.655 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:33.663 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:33.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:33.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:33.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:34.004 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-13 zones={test-zone3, test-zone1} 2021-02-08T07:19:34.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:34.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:34.005 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:34.041 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:34.042 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:34.279 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.22.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:34.280 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:34.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:34.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:34.280 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:34.325 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:34.586 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.22.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:34.587 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:34.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:34.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:34.587 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:34.837 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:34.837 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-14 2021-02-08T07:19:34.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:34.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:34.838 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:34.885 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:35.142 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:35.143 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-14 2021-02-08T07:19:35.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-14 zones={test-zone1, test-zone2} 2021-02-08T07:19:36.631 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:36.640 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:36.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:36.687 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:36.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:36.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:36.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:37.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-14 zones={test-zone1, test-zone2} 2021-02-08T07:19:37.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:37.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:37.019 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:37.057 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:37.057 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:37.270 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:37.270 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-14 2021-02-08T07:19:37.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:37.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:37.271 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:37.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:37.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:37.559 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:37.559 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-14 2021-02-08T07:19:37.559 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-14 zones={test-zone3, test-zone2} 2021-02-08T07:19:38.083 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:38.135 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:38.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:38.142 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:38.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:38.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:38.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:38.481 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-14 zones={test-zone3, test-zone2} 2021-02-08T07:19:38.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:38.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:38.482 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:38.525 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:38.754 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:38.754 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-15 2021-02-08T07:19:38.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:38.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:38.755 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:38.839 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:38.840 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:39.107 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:39.108 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-15 2021-02-08T07:19:39.108 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-15 zones={test-zone1, test-zone3} 2021-02-08T07:19:40.502 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:40.509 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:40.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:40.563 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:40.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:40.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:40.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:40.906 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-15 zones={test-zone1, test-zone3} 2021-02-08T07:19:40.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:40.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:40.906 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:40.945 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:40.945 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:41.185 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:41.185 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-15 2021-02-08T07:19:41.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:41.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:41.186 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:41.232 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:41.232 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:41.464 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:41.465 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-15 2021-02-08T07:19:41.465 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-15 zones={test-zone2, test-zone3} 2021-02-08T07:19:42.582 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:42.588 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:42.588 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:42.638 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-08T07:19:42.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:42.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:42.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-08T07:19:42.983 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-15 zones={test-zone2, test-zone3} 2021-02-08T07:19:43.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:43.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:43.627 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:43.671 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:43.672 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:43.913 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.24.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:43.913 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:43.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:43.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:43.914 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:43.960 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:43.960 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:44.182 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.22.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:44.182 WARNING:rgw_multi.tests:bucket lhddgo-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.24.5 target=00000000001.22.5 2021-02-08T07:19:49.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:49.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:49.183 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:49.225 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:49.226 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:49.449 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.24.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:49.450 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:49.450 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-13 zones={test-zone2, test-zone1} 2021-02-08T07:19:50.275 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:50.282 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:50.386 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-13 zones={test-zone2, test-zone1} 2021-02-08T07:19:50.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:50.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:50.387 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:50.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:50.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:50.655 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.24.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:50.655 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:50.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:50.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:50.656 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:50.738 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:50.739 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:50.979 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.24.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:50.979 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:50.979 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-13 zones={test-zone3, test-zone1} 2021-02-08T07:19:52.209 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:52.216 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:52.323 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-13 zones={test-zone3, test-zone1} 2021-02-08T07:19:52.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:52.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:52.323 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:52.365 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:52.365 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:52.596 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.24.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:52.596 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:52.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:52.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:52.597 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:52.647 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:52.648 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:52.909 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.24.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:52.910 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-13 2021-02-08T07:19:53.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:53.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:53.899 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:53.936 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:53.936 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:54.151 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.22.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:54.151 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-14 2021-02-08T07:19:54.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:54.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:54.152 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:54.200 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:54.200 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:54.436 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.22.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:54.437 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-14 2021-02-08T07:19:54.437 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-14 zones={test-zone1, test-zone2} 2021-02-08T07:19:56.132 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:56.141 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:56.254 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-14 zones={test-zone1, test-zone2} 2021-02-08T07:19:56.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:56.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:56.255 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:56.294 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:56.295 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:56.519 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.22.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:56.519 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-14 2021-02-08T07:19:56.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:56.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:56.520 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:56.563 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:56.564 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:56.805 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.22.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:56.805 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-14 2021-02-08T07:19:56.806 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-14 zones={test-zone3, test-zone2} 2021-02-08T07:19:57.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:19:57.932 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:19:58.034 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-14 zones={test-zone3, test-zone2} 2021-02-08T07:19:58.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:58.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:58.055 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:58.093 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:58.094 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:58.323 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.23.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:19:58.324 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-15 2021-02-08T07:19:58.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:58.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:19:58.324 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:19:58.408 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:19:58.408 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:19:58.669 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:19:58.670 WARNING:rgw_multi.tests:bucket lhddgo-15 zone test-zone1 behind zone test-zone3: shard=1 master=00000000002.23.5 target=00000000001.11.5 2021-02-08T07:20:03.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:03.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:03.671 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:03.714 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:03.714 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:03.974 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:03.974 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-15 2021-02-08T07:20:03.974 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-15 zones={test-zone1, test-zone3} 2021-02-08T07:20:04.998 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:20:09.324 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:20:09.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-15 zones={test-zone1, test-zone3} 2021-02-08T07:20:09.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:09.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:09.849 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:09.895 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:09.895 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:10.133 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.23.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:10.133 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-15 2021-02-08T07:20:10.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:10.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:10.134 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:10.181 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:10.181 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:10.416 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:10.416 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-15 2021-02-08T07:20:10.416 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-15 zones={test-zone2, test-zone3} 2021-02-08T07:20:11.328 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:20:11.338 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:20:11.456 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-15 zones={test-zone2, test-zone3} 2021-02-08T07:20:11.457 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2021-02-08T07:20:11.457 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-16 2021-02-08T07:20:12.022 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-17 2021-02-08T07:20:12.078 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-18 2021-02-08T07:20:12.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:12.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:12.178 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:12.213 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:12.213 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:12.414 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:20:12.455 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612768812.172314_175.1","last_update":"2021-02-08T07:20:12.172314Z"}] 2021-02-08T07:20:12.455 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:20:12.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:12.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:12.456 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:12.496 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:12.713 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768763.570467_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:19:23.570467Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:20:12.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612768812.172314_175.1 target=1_1612768763.570467_145.1 2021-02-08T07:20:17.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:17.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:17.715 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:17.756 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:17.756 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:17.964 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768812.172314_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:20:12.172314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:20:17.965 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:20:17.965 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:20:17.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:17.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:17.966 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:18.005 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:18.005 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:18.226 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768812.172314_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:20:12.172314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:20:18.226 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:20:18.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:20:18.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:18.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:18.227 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:18.265 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:18.265 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:18.535 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768812.172314_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:20:12.172314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:20:18.535 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:20:19.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:19.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:19.527 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:19.567 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:19.567 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:19.798 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.18.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:19.799 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:19.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:19.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:19.799 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:19.844 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:19.844 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:20.069 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:20.069 WARNING:rgw_multi.tests:bucket lhddgo-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.18.10 target= 2021-02-08T07:20:25.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:25.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:25.071 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:25.117 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:25.117 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:25.348 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000005.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:25.349 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:25.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:25.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:25.350 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:25.389 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:25.390 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:25.625 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:25.626 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:25.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:25.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:25.626 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:25.749 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:25.749 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:26.003 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000005.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:26.003 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:26.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:26.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:26.004 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:26.085 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:26.326 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:26.326 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:26.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:26.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:26.327 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:26.369 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:26.370 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:26.643 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:26.644 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:27.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:27.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:27.003 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:27.044 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:27.044 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:27.258 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.13.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:27.259 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:20:27.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:27.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:27.259 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:27.305 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:27.306 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:27.554 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:27.554 WARNING:rgw_multi.tests:bucket lhddgo-17 zone test-zone1 behind zone test-zone2: shard=5 master=00000000002.13.10 target= 2021-02-08T07:20:32.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:32.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:32.556 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:32.608 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:32.608 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:32.850 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000005.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:32.851 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:20:32.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:32.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:32.851 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:32.886 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:32.886 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:33.096 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:33.097 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:20:33.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:33.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:33.097 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:33.142 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:33.142 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:33.395 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000005.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:33.396 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:20:34.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:34.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:34.347 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:34.387 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:34.387 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:34.620 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.26.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:34.620 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:20:34.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:34.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:34.621 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:34.706 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:34.706 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:34.957 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:34.957 WARNING:rgw_multi.tests:bucket lhddgo-18 zone test-zone1 behind zone test-zone3: shard=5 master=00000000002.26.10 target= 2021-02-08T07:20:39.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:39.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:39.959 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:40.004 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:40.004 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:40.260 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000006.33.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:40.261 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:20:40.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:40.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:40.261 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:40.343 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:40.343 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:40.579 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.33.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:40.580 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:20:40.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:40.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:40.580 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:40.625 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:40.625 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:40.844 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000006.33.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:40.844 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:20:46.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:46.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:46.543 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:46.582 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:46.582 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:46.814 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.28.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.32.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.38.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:46.814 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:46.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:46.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:46.815 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:46.860 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:46.861 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:47.071 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000010.28.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000003.32.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000008.38.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:47.072 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:47.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:47.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:47.073 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:47.123 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:47.124 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:47.351 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.28.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.32.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.38.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:47.351 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:47.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:47.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:47.352 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:47.397 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:47.397 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:47.640 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000010.28.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000003.32.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000008.38.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:47.643 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:47.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:47.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:47.643 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:47.750 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:47.751 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:47.995 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.28.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.32.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.38.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:47.995 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:47.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:47.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:47.996 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:48.041 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:48.041 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:48.308 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.28.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.32.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.38.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:48.309 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:20:48.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:48.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:48.309 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:48.611 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.44.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.34.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:48.612 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-16 2021-02-08T07:20:48.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:48.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:48.612 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:48.656 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:48.656 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:48.901 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000010.44.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000007.34.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:48.901 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-16 2021-02-08T07:20:48.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:48.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:48.902 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:48.947 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:48.947 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:49.155 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.44.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.34.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:49.155 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-16 2021-02-08T07:20:49.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:49.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:49.155 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:49.210 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:49.211 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:49.480 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000010.44.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000007.34.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:49.481 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-16 2021-02-08T07:20:49.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:49.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:49.481 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:49.567 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:49.568 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:49.822 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.33.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:49.823 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-16 2021-02-08T07:20:49.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:49.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:49.823 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:49.911 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:49.911 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:50.158 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000003.33.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000014.54.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:50.158 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-16 2021-02-08T07:20:50.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:50.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:50.159 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:50.200 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:50.200 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:50.448 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.33.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:50.449 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-16 2021-02-08T07:20:50.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:50.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:50.449 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:50.492 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:50.493 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:50.706 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000003.33.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000014.54.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:50.706 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-16 2021-02-08T07:20:50.706 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-16 zones={test-zone2, test-zone1} 2021-02-08T07:20:51.386 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:20:51.394 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:20:51.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:20:51.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:20:51.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:20:51.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:20:51.405 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:20:51.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:20:51.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:20:51.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:20:51.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:20:51.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:20:51.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:20:51.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:20:51.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:20:51.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:20:51.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:20:51.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:20:52.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:20:52.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:20:52.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:20:52.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:20:52.432 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-16 zones={test-zone2, test-zone1} 2021-02-08T07:20:52.432 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-16 zones={test-zone3, test-zone1} 2021-02-08T07:20:52.553 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:20:52.559 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:20:52.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:20:52.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:20:52.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:20:52.559 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:20:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:20:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:20:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:20:52.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:20:52.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:20:52.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:20:52.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:20:52.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:20:52.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:20:52.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:20:53.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:20:53.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:20:53.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:20:53.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:20:53.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:20:53.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:20:53.569 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-16 zones={test-zone3, test-zone1} 2021-02-08T07:20:53.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-16 zones={test-zone3, test-zone2} 2021-02-08T07:20:54.306 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:20:54.317 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:20:54.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:20:54.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:20:54.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:20:54.317 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:20:54.324 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:20:54.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:20:54.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:20:54.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:20:54.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:20:54.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:20:54.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:20:54.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:20:54.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:20:54.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:20:54.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:20:54.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:20:54.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:20:55.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:20:55.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:20:55.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:20:55.330 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-16 zones={test-zone3, test-zone2} 2021-02-08T07:20:55.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:55.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:55.331 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:55.374 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:55.374 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:55.621 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.24.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:55.621 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:20:55.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:55.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:55.622 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:55.897 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Z6B20eTjlDJ3aCZ4rwDZtGl-oLR5vaj","ns":""},"count":1},"inc_marker":{"position":"00000000010.34.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000007.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:55.897 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:20:55.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:55.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:55.898 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:55.940 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:55.940 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:56.193 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.24.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:56.193 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:20:56.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:56.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:56.194 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:56.278 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:56.279 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:56.581 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Z6B20eTjlDJ3aCZ4rwDZtGl-oLR5vaj","ns":""},"count":1},"inc_marker":{"position":"00000000010.34.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000007.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:56.582 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:20:56.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:56.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:56.582 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:56.626 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:56.626 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:56.890 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.24.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:56.890 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:20:56.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:56.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:56.891 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:57.207 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.34.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.24.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:57.207 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:20:57.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:57.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:57.208 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:57.289 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:57.289 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:57.512 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.16.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.37.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:57.512 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:20:57.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:57.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:57.513 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:57.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:57.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:57.832 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"WjcBB9jFEUiDkYHsbAYwtmr8GHqGddS","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":2},"inc_marker":{"position":"00000000010.37.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000008.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:57.833 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:20:57.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:57.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:57.833 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:57.875 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:57.875 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:58.089 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.16.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.37.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:58.089 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:20:58.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:58.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:58.090 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:58.133 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:58.133 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:58.404 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"WjcBB9jFEUiDkYHsbAYwtmr8GHqGddS","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":2},"inc_marker":{"position":"00000000010.37.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000008.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:58.405 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:20:58.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:58.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:58.405 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:58.488 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:58.488 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:58.731 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.21.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:58.732 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-17 2021-02-08T07:20:58.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:58.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:58.732 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:58.818 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:58.819 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:59.105 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"WjcBB9jFEUiDkYHsbAYwtmr8GHqGddS","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000014.34.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:59.106 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-17 2021-02-08T07:20:59.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:59.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:59.107 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:59.145 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:59.399 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.21.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:20:59.400 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-17 2021-02-08T07:20:59.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:59.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:20:59.400 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:20:59.445 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:20:59.445 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:20:59.673 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"WjcBB9jFEUiDkYHsbAYwtmr8GHqGddS","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000014.34.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:20:59.673 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-17 2021-02-08T07:20:59.673 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-17 zones={test-zone2, test-zone1} 2021-02-08T07:21:01.480 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:01.488 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:01.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:01.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:01.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:01.488 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:01.494 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:01.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:01.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:01.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:01.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:01.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:01.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:01.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:01.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:01.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:01.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:01.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:02.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:02.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:02.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:02.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:02.507 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-17 zones={test-zone2, test-zone1} 2021-02-08T07:21:02.507 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-17 zones={test-zone3, test-zone1} 2021-02-08T07:21:03.507 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:03.513 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:03.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:03.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:03.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:03.513 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:03.519 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:03.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:03.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:03.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:03.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:03.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:03.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:03.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:03.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:03.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:03.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:04.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:04.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:04.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:04.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:04.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:04.503 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-17 zones={test-zone3, test-zone1} 2021-02-08T07:21:04.503 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-17 zones={test-zone3, test-zone2} 2021-02-08T07:21:04.942 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:04.952 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:04.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:04.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:04.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:04.952 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:04.958 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:04.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:04.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:04.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:04.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:04.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:05.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:05.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:05.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:05.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:05.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:05.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:05.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:05.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:05.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:05.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:05.957 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-17 zones={test-zone3, test-zone2} 2021-02-08T07:21:05.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:05.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:05.958 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:06.001 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:06.001 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:06.246 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.44.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:06.247 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:06.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:06.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:06.247 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:06.530 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":2},"inc_marker":{"position":"00000000010.25.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000007.44.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:06.531 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:06.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:06.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:06.531 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:06.575 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:06.575 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:06.795 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.44.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:06.795 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:06.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:06.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:06.796 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:06.840 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:06.840 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:07.121 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000010.25.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000007.44.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:07.259 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:07.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:07.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:07.260 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:07.313 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:07.314 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:07.567 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.44.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:07.567 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:07.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:07.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:07.568 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:07.615 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:07.615 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:07.879 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.25.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.44.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:07.879 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:07.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:07.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:07.880 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:07.919 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:07.919 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:08.129 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.28.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.40.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:08.129 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-18 2021-02-08T07:21:08.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:08.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:08.130 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:08.172 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:08.173 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:08.445 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000010.28.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000007.40.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:08.446 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-18 2021-02-08T07:21:08.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:08.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:08.446 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:08.488 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:08.488 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:08.688 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.28.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.40.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:08.689 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-18 2021-02-08T07:21:08.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:08.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:08.689 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:08.734 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:08.734 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:09.026 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000010.28.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000007.40.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:09.026 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-18 2021-02-08T07:21:09.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:09.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:09.027 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:09.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:09.109 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:09.366 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.17.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.14.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.74.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:09.366 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:21:09.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:09.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:09.367 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:09.450 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:09.715 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000016.74.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:09.715 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:21:09.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:09.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:09.716 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:09.801 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:09.802 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:10.060 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.17.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.14.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.74.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:10.060 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:21:10.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:10.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:10.061 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:10.106 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:10.106 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:10.354 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000016.74.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:10.354 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:21:10.355 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-18 zones={test-zone2, test-zone1} 2021-02-08T07:21:11.535 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:11.547 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:11.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:11.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:11.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:11.547 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:11.558 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:11.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:11.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:11.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:11.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:11.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:11.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:11.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:11.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:11.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:12.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:12.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:12.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:12.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:12.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:12.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:12.705 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-18 zones={test-zone2, test-zone1} 2021-02-08T07:21:12.706 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-18 zones={test-zone3, test-zone1} 2021-02-08T07:21:12.969 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:12.977 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:12.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:12.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:12.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:12.978 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:12.984 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:12.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:12.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:12.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:13.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:13.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:13.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:13.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:13.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:13.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:13.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:13.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:13.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:13.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:13.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:13.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:14.024 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-18 zones={test-zone3, test-zone1} 2021-02-08T07:21:14.025 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-18 zones={test-zone3, test-zone2} 2021-02-08T07:21:14.311 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:14.321 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:14.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:14.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:14.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:14.322 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:14.328 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:14.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:14.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:14.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:14.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:14.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:14.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:14.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:14.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:14.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:14.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:14.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:14.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:15.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:15.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:15.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:15.306 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-18 zones={test-zone3, test-zone2} 2021-02-08T07:21:16.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:16.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:16.406 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:16.446 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:16.446 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:16.708 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.32.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.38.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:16.708 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:21:16.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:16.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:16.709 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:16.987 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000010.28.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000003.32.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000008.38.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:16.988 WARNING:rgw_multi.tests:bucket lhddgo-16 zone test-zone2 behind zone test-zone1: shard=1 master=00000000011.34.5 target=00000000010.28.5 2021-02-08T07:21:21.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:21.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:21.989 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:22.031 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:22.031 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:22.274 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000011.34.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000004.35.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000009.47.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:22.274 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:21:22.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:22.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:22.275 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:22.318 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:22.563 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.35.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:22.563 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:21:22.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:22.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:22.564 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:22.651 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:22.651 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:22.913 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000011.34.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000004.35.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000009.47.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:22.913 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:21:22.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:22.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:22.914 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:22.954 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:22.954 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:23.199 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.35.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:23.200 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:21:23.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:23.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:23.200 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:23.246 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:23.247 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:23.523 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.34.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.47.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:23.524 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-16 2021-02-08T07:21:23.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:23.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:23.524 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:23.564 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:23.564 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:23.776 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.50.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.43.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:23.776 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-16 2021-02-08T07:21:23.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:23.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:23.777 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:23.827 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:23.827 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:24.091 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000011.50.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000008.43.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:24.091 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-16 2021-02-08T07:21:24.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:24.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:24.091 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:24.133 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:24.134 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:24.333 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.50.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.43.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:24.333 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-16 2021-02-08T07:21:24.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:24.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:24.334 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:24.380 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:24.380 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:24.641 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000011.50.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000008.43.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:24.642 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-16 2021-02-08T07:21:24.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:24.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:24.643 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:24.684 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:24.684 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:24.926 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.77.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:24.927 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-16 2021-02-08T07:21:24.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:24.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:24.927 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:25.015 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:25.015 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:25.263 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000004.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000015.77.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:25.263 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-16 2021-02-08T07:21:25.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:25.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:25.264 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:25.305 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:25.305 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:25.555 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.77.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:25.555 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-16 2021-02-08T07:21:25.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:25.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:25.556 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:25.605 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:25.605 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:25.825 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"VzdN-Ad9uGUXQJw649B48pRZOAyHjik","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"OaPMVKb8QQ4PcAPj9UWDXBNaBiSzVPv","ns":""},"count":1},"inc_marker":{"position":"00000000004.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"VrC4Y.17Sj5AmWehSa0OiwOmKIK3gsc","ns":""},"count":1},"inc_marker":{"position":"00000000015.77.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:25.825 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-16 2021-02-08T07:21:25.825 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-16 zones={test-zone2, test-zone1} 2021-02-08T07:21:27.089 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:27.097 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:27.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:27.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:27.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:27.097 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:27.105 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:27.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:27.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:27.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:27.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:27.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:27.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:27.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:27.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:27.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:27.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:27.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:27.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:27.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:27.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:28.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:28.120 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-16 zones={test-zone2, test-zone1} 2021-02-08T07:21:28.120 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-16 zones={test-zone3, test-zone1} 2021-02-08T07:21:28.506 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:28.529 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:28.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:28.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:28.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:28.532 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:28.547 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:28.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:28.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:28.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:28.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:28.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:28.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:28.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:28.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:28.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:29.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:29.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:29.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:29.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:29.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:29.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:29.604 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-16 zones={test-zone3, test-zone1} 2021-02-08T07:21:29.605 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-16 zones={test-zone3, test-zone2} 2021-02-08T07:21:30.534 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:30.545 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:30.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:30.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:30.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:30.546 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:30.554 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:30.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:30.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:30.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:30.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:30.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:30.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:30.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:30.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:30.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:30.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:31.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:31.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:31.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:31.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:31.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:31.562 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-16 zones={test-zone3, test-zone2} 2021-02-08T07:21:31.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:31.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:31.563 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:31.604 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:31.604 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:31.835 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.40.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.27.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.27.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:31.835 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:21:31.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:31.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:31.835 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:31.880 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:31.881 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:32.123 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Z6B20eTjlDJ3aCZ4rwDZtGl-oLR5vaj","ns":""},"count":1},"inc_marker":{"position":"00000000011.40.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":1},"inc_marker":{"position":"00000000004.27.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000008.27.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:32.124 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:21:32.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:32.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:32.124 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:32.166 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:32.167 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:32.412 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.40.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.27.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.27.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:32.412 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:21:32.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:32.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:32.413 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:32.468 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:32.468 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:32.725 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Z6B20eTjlDJ3aCZ4rwDZtGl-oLR5vaj","ns":""},"count":1},"inc_marker":{"position":"00000000011.40.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":1},"inc_marker":{"position":"00000000004.27.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000008.27.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:32.726 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:21:32.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:32.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:32.726 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:32.839 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:32.839 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:33.087 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.40.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.27.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.27.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:33.088 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:21:33.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:33.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:33.088 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:33.136 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:33.136 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:33.402 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.40.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.27.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.27.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:33.403 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-17 2021-02-08T07:21:33.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:33.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:33.403 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:33.721 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.43.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.28.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:33.723 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:21:33.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:33.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:33.724 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:33.777 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:33.777 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:34.051 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"WjcBB9jFEUiDkYHsbAYwtmr8GHqGddS","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":2},"inc_marker":{"position":"00000000011.43.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:34.052 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:21:34.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:34.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:34.052 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:34.093 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:34.094 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:34.313 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.43.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.28.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:34.313 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:21:34.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:34.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:34.314 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:34.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:34.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:34.626 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"WjcBB9jFEUiDkYHsbAYwtmr8GHqGddS","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":2},"inc_marker":{"position":"00000000011.43.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:34.626 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-17 2021-02-08T07:21:34.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:34.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:34.627 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:34.707 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:34.707 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:34.936 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.24.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:34.936 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-17 2021-02-08T07:21:34.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:34.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:34.937 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:35.022 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:35.282 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"WjcBB9jFEUiDkYHsbAYwtmr8GHqGddS","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000015.37.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:35.282 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-17 2021-02-08T07:21:35.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:35.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:35.283 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:35.363 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:35.363 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:35.610 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.24.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:35.610 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-17 2021-02-08T07:21:35.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:35.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:35.611 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:35.652 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:35.652 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:35.889 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"WjcBB9jFEUiDkYHsbAYwtmr8GHqGddS","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"P0u5d.kfCSJ.4QvfejT2paSj2mjPghW","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"-VtmfRvzYf5lnSqL-RcY0SyS4bP1Ixb","ns":""},"count":1},"inc_marker":{"position":"00000000015.37.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:35.890 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-17 2021-02-08T07:21:35.890 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-17 zones={test-zone2, test-zone1} 2021-02-08T07:21:36.502 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:36.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:36.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:36.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:36.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:36.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:36.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:36.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:36.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:36.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:37.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:37.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:37.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:37.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:37.495 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-17 zones={test-zone2, test-zone1} 2021-02-08T07:21:37.495 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-17 zones={test-zone3, test-zone1} 2021-02-08T07:21:37.634 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:37.639 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:37.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:37.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:37.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:37.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:37.645 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:37.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:37.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:37.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:37.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:37.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:37.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:37.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:37.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:37.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:38.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:38.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:38.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:38.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:38.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:38.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:38.638 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-17 zones={test-zone3, test-zone1} 2021-02-08T07:21:38.638 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-17 zones={test-zone3, test-zone2} 2021-02-08T07:21:39.075 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:39.085 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:39.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:39.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:39.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:39.085 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:39.092 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:39.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:39.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:39.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:39.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:39.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:39.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:39.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:39.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:39.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:39.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:39.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:39.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:39.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:39.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:39.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:40.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-17 zones={test-zone3, test-zone2} 2021-02-08T07:21:40.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:40.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:40.077 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:40.120 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:40.120 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:40.372 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.49.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:40.373 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:40.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:40.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:40.373 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:40.417 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:40.418 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:40.650 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":2},"inc_marker":{"position":"00000000011.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000008.49.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:40.651 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:40.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:40.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:40.652 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:40.695 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:40.695 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:40.938 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.49.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:40.939 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:40.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:40.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:40.939 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:41.022 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:41.022 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:41.294 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000011.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000008.49.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:41.295 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:41.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:41.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:41.295 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:41.382 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:41.383 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:41.621 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.49.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:41.622 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:41.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:41.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:41.622 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:41.664 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:41.664 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:41.932 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.49.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:41.933 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-18 2021-02-08T07:21:41.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:41.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:41.933 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:42.014 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:42.014 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:42.235 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:42.235 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-18 2021-02-08T07:21:42.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:42.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:42.236 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:42.280 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:42.281 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:42.546 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000011.34.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000008.45.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:42.546 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-18 2021-02-08T07:21:42.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:42.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:42.547 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:42.588 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:42.588 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:42.797 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:42.797 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-18 2021-02-08T07:21:42.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:42.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:42.798 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:42.843 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:42.843 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:43.098 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000011.34.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000008.45.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:43.098 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-18 2021-02-08T07:21:43.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:43.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:43.099 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:43.180 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:43.180 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:43.413 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.20.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.17.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.79.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:43.413 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:21:43.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:43.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:43.414 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:43.498 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:43.499 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:43.769 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000017.79.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:43.769 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:21:43.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:43.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:43.770 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:43.853 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:43.853 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:44.091 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.20.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.17.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.79.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:44.091 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:21:44.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:44.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:44.091 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:44.135 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:44.135 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:44.369 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7.zxdTW-OHZIrIe71mgG5m1E7-RBf6Y","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"tENrURnv.z3qLOLya4jv3-T5Mw84YOS","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5txqGc9F3HBXEJUIkSCf3sLrX7bbCev","ns":""},"count":1},"inc_marker":{"position":"00000000017.79.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:44.370 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-18 2021-02-08T07:21:44.370 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-18 zones={test-zone2, test-zone1} 2021-02-08T07:21:44.912 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:44.919 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:44.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:44.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:44.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:44.919 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:44.926 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:44.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:44.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:44.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:44.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:44.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:45.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:45.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:45.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:45.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:45.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:45.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:45.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:45.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:45.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:45.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:45.916 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-18 zones={test-zone2, test-zone1} 2021-02-08T07:21:45.917 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-18 zones={test-zone3, test-zone1} 2021-02-08T07:21:46.229 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:46.235 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:46.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:46.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:46.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:46.235 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:46.240 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:46.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:46.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:46.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:46.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:46.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:46.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:46.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:46.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:46.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:46.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:46.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:46.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:46.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:46.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:47.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:47.223 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-18 zones={test-zone3, test-zone1} 2021-02-08T07:21:47.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-18 zones={test-zone3, test-zone2} 2021-02-08T07:21:48.219 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:21:48.229 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:48.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:48.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:48.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:48.230 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:21:48.235 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:21:48.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-08T07:21:48.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-08T07:21:48.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-08T07:21:48.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:48.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:48.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:21:48.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:48.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:48.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-08T07:21:48.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:48.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:48.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-08T07:21:48.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:49.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:49.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-08T07:21:49.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-18 zones={test-zone3, test-zone2} 2021-02-08T07:21:49.274 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-02-08T07:21:49.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:49.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:49.782 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:49.818 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:49.818 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:50.021 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:21:50.059 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612768909.778782_219.1","last_update":"2021-02-08T07:21:49.778782Z"}] 2021-02-08T07:21:50.059 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:21:50.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:50.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:50.060 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:50.096 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:50.303 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768812.172314_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:20:12.172314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:21:50.303 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612768909.778782_219.1 target=1_1612768812.172314_175.1 2021-02-08T07:21:55.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:55.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:55.305 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:55.346 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:55.347 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:55.570 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768909.778782_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:49.778782Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:21:55.570 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:21:55.570 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:21:55.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:55.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:55.571 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:55.616 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:55.616 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:55.865 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768909.778782_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:49.778782Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:21:55.866 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:21:55.866 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:21:55.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:55.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:55.866 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:55.908 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:55.908 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:56.189 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768909.778782_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:49.778782Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:21:56.189 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:21:56.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:56.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:56.667 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:56.709 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:56.709 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:56.958 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.19.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:21:56.958 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:21:56.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:56.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:21:56.959 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:21:57.004 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:21:57.004 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:21:57.234 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:21:57.234 WARNING:rgw_multi.tests:bucket lhddgo-19 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.19.10 target= 2021-02-08T07:22:02.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:02.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:02.236 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:02.279 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:02.280 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:02.507 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000005.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:02.507 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:02.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:02.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:02.508 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:02.548 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:02.549 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:02.795 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:02.795 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:02.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:02.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:02.796 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:02.888 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:02.888 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:03.154 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000005.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:03.155 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:03.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:03.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:03.155 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:03.238 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:03.238 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:03.484 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:03.485 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:03.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:03.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:03.485 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:03.528 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:03.528 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:03.801 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:03.801 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:03.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:03.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:03.802 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:03.883 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:03.884 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:04.098 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:04.099 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-19 2021-02-08T07:22:04.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:04.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:04.099 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:04.145 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:04.145 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:04.419 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:04.419 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-19 2021-02-08T07:22:04.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:04.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:04.420 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:04.462 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:04.462 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:04.680 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:04.680 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-19 2021-02-08T07:22:04.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:04.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:04.681 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:04.730 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:04.730 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:04.989 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:04.990 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-19 2021-02-08T07:22:04.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:04.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:04.990 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:05.029 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:05.029 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:05.268 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:05.268 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-19 2021-02-08T07:22:05.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:05.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:05.268 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:05.352 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:05.353 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:05.615 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:05.615 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-19 2021-02-08T07:22:05.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:05.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:05.616 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:05.696 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:05.696 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:05.941 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:05.941 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-19 2021-02-08T07:22:05.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:05.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:05.942 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:05.985 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:05.986 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:06.225 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:06.225 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-19 2021-02-08T07:22:06.225 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-19 zones={test-zone2, test-zone1} 2021-02-08T07:22:07.898 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:22:07.904 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:07.904 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:22:07.951 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:08.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:08.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:08.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:08.156 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-19 zones={test-zone2, test-zone1} 2021-02-08T07:22:08.156 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-19 zones={test-zone3, test-zone1} 2021-02-08T07:22:08.842 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:22:08.887 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:08.887 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:22:08.894 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:08.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:08.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:09.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:09.091 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-19 zones={test-zone3, test-zone1} 2021-02-08T07:22:09.091 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-19 zones={test-zone3, test-zone2} 2021-02-08T07:22:09.101 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:22:09.150 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:09.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:22:09.157 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:09.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:09.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:09.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:09.348 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-19 zones={test-zone3, test-zone2} 2021-02-08T07:22:11.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:11.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:11.135 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:11.177 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:11.178 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:11.389 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000043.134.8"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:11.389 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:11.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:11.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:11.390 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:11.434 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:11.434 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:11.643 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000005.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"uTGEOABhONVlEc09VXsT-itMuq9H41y","ns":""},"count":1},"inc_marker":{"position":"00000000042.133.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:11.644 WARNING:rgw_multi.tests:bucket lhddgo-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000043.134.8 target=00000000042.133.5 2021-02-08T07:22:16.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:16.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:16.645 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:16.690 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:16.691 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:16.922 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000005.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"uTGEOABhONVlEc09VXsT-itMuq9H41y","ns":""},"count":1},"inc_marker":{"position":"00000000047.144.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:16.923 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:16.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:16.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:16.923 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:16.964 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:16.964 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:17.205 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.144.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:17.205 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:17.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:17.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:17.206 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:17.293 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:17.545 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000005.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"uTGEOABhONVlEc09VXsT-itMuq9H41y","ns":""},"count":1},"inc_marker":{"position":"00000000047.144.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:17.546 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:17.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:17.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:17.546 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:17.583 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:17.583 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:17.814 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.144.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:17.814 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:17.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:17.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:17.815 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:17.862 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:17.862 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:18.135 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000047.144.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:18.136 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-19 2021-02-08T07:22:18.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:18.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:18.136 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:18.445 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000049.169.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:18.445 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-19 2021-02-08T07:22:18.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:18.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:18.446 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:18.493 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:18.493 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:18.760 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"tEAfXfSOs1.UOVMOI39DXCpeRGExuJA","ns":""},"count":6},"inc_marker":{"position":"00000000049.169.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:18.761 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-19 2021-02-08T07:22:18.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:18.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:18.761 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:18.806 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:18.807 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:19.033 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000049.169.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:19.034 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-19 2021-02-08T07:22:19.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:19.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:19.034 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:19.082 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:19.082 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:19.342 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"tEAfXfSOs1.UOVMOI39DXCpeRGExuJA","ns":""},"count":6},"inc_marker":{"position":"00000000049.169.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:19.343 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-19 2021-02-08T07:22:19.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:19.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:19.343 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:19.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:19.430 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:19.670 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000043.150.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:19.670 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-19 2021-02-08T07:22:19.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:19.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:19.671 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:19.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:19.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:20.034 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"RCa21VPT3YcTQ3wd0Nyh.FF2QFSAKmT","ns":""},"count":1},"inc_marker":{"position":"00000000043.150.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:20.034 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-19 2021-02-08T07:22:20.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:20.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:20.035 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:20.075 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:20.075 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:20.318 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000043.150.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:22:20.319 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-19 2021-02-08T07:22:20.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:20.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:20.319 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:20.364 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:20.365 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:20.610 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"93jlK5MXPOkRuPrag.lSuMMrduOPihm","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"RCa21VPT3YcTQ3wd0Nyh.FF2QFSAKmT","ns":""},"count":1},"inc_marker":{"position":"00000000043.150.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:22:20.610 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-19 2021-02-08T07:22:20.610 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-19 zones={test-zone2, test-zone1} 2021-02-08T07:22:21.747 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.761 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:22:21.774 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:21.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:21.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:21.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:21.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:21.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:21.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:22.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:22.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:22.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:22.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:22.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:22.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:22.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:22.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:22.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:23.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:24.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:25.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:26.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:27.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:28.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:28.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:28.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:28.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:28.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:28.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:28.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:28.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:28.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:29.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:30.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-19 zones={test-zone2, test-zone1} 2021-02-08T07:22:30.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-19 zones={test-zone3, test-zone1} 2021-02-08T07:22:31.520 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:22:31.527 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:31.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.530 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:31.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:31.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:31.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:31.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:31.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:31.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:31.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:32.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:33.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:34.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:35.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:36.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:37.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:38.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:39.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:40.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:40.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:40.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:40.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:40.476 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-19 zones={test-zone3, test-zone1} 2021-02-08T07:22:40.476 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-19 zones={test-zone3, test-zone2} 2021-02-08T07:22:41.140 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:22:41.151 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:41.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.153 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:22:41.164 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-08T07:22:41.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:22:41.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:41.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:41.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-08T07:22:41.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:41.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:41.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:41.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:41.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:41.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:41.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:41.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:42.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:43.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:44.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:45.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:46.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:47.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:48.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:48.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:48.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:48.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:48.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:48.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:48.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:48.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:48.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:49.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:49.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:49.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:49.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:49.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:49.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:49.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:49.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:49.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:50.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:22:50.216 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-19 zones={test-zone3, test-zone2} 2021-02-08T07:22:50.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:50.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:50.216 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:50.259 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:50.259 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:50.527 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1612768880.006161_251.3","last_update":"2021-02-08T07:21:20.006161Z"},{"marker":"1_1612768946.017965_277.1","last_update":"2021-02-08T07:22:26.017965Z"},{"marker":"1_1612768924.012632_160.1","last_update":"2021-02-08T07:22:04.012632Z"}] 2021-02-08T07:22:50.527 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-08T07:22:50.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:50.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:50.527 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:50.571 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:50.571 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:50.799 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612768946.017965_277.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:26.017965Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.012632_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.012632Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:22:50.799 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-08T07:22:50.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:50.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:50.800 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:50.839 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:50.839 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:51.095 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1612768880.006161_251.3","last_update":"2021-02-08T07:21:20.006161Z"},{"marker":"1_1612768946.017965_277.1","last_update":"2021-02-08T07:22:26.017965Z"},{"marker":"1_1612768924.012632_160.1","last_update":"2021-02-08T07:22:04.012632Z"}] 2021-02-08T07:22:51.095 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-08T07:22:51.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:51.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:51.096 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:51.181 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:51.181 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:51.430 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7430472880135713257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612768946.017965_277.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:26.017965Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.012632_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.012632Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:22:51.430 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-08T07:22:51.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:51.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:51.431 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:51.510 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:51.510 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:51.722 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1612768880.006161_251.3","last_update":"2021-02-08T07:21:20.006161Z"},{"marker":"1_1612768946.017965_277.1","last_update":"2021-02-08T07:22:26.017965Z"},{"marker":"1_1612768924.012632_160.1","last_update":"2021-02-08T07:22:04.012632Z"}] 2021-02-08T07:22:51.723 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-08T07:22:51.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:51.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:51.723 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:51.765 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:51.766 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:52.037 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":729464676215090460},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612768946.017965_277.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:26.017965Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.012632_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.012632Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:22:52.037 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-08T07:22:52.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:52.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:52.038 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:52.119 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:52.119 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:52.342 INFO:teuthology.orchestra.run.smithi110.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1612768881.007212_292.3","last_update":"2021-02-08T07:21:21.007212Z"},{"marker":"1_1612768947.021142_296.1","last_update":"2021-02-08T07:22:27.021142Z"},{"marker":"1_1612768925.016683_163.1","last_update":"2021-02-08T07:22:05.016683Z"}] 2021-02-08T07:22:52.342 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-08T07:22:52.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:52.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:52.343 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:52.385 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:52.385 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:52.654 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612768947.021142_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:27.021142Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768925.016683_163.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:05.016683Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:22:52.654 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-08T07:22:52.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:52.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:52.655 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:52.695 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:52.695 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:52.920 INFO:teuthology.orchestra.run.smithi110.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1612768881.007212_292.3","last_update":"2021-02-08T07:21:21.007212Z"},{"marker":"1_1612768947.021142_296.1","last_update":"2021-02-08T07:22:27.021142Z"},{"marker":"1_1612768925.016683_163.1","last_update":"2021-02-08T07:22:05.016683Z"}] 2021-02-08T07:22:52.920 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-08T07:22:52.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:52.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:52.921 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:52.964 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:52.964 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:53.213 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":711522086589135958},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612768947.021142_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:27.021142Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768925.016683_163.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:05.016683Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:22:53.213 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-08T07:22:53.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:53.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:53.214 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:53.295 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:53.296 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:53.547 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1612768880.010730_219.3","last_update":"2021-02-08T07:21:20.010730Z"},{"marker":"1_1612768946.023049_235.1","last_update":"2021-02-08T07:22:26.023049Z"},{"marker":"1_1612768924.017986_97.1","last_update":"2021-02-08T07:22:04.017986Z"}] 2021-02-08T07:22:53.547 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-08T07:22:53.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:53.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:53.548 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:53.629 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:53.883 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12652603110286941624},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.010730_219.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.010730Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612768946.023049_235.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:26.023049Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.017986_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.017986Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:22:53.883 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-08T07:22:53.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:53.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:53.884 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:53.964 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:53.964 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:54.208 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1612768880.010730_219.3","last_update":"2021-02-08T07:21:20.010730Z"},{"marker":"1_1612768946.023049_235.1","last_update":"2021-02-08T07:22:26.023049Z"},{"marker":"1_1612768924.017986_97.1","last_update":"2021-02-08T07:22:04.017986Z"}] 2021-02-08T07:22:54.208 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-08T07:22:54.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:54.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:54.209 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:54.250 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:54.250 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:54.477 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5378454525578809946},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.010730_219.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.010730Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612768946.023049_235.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:26.023049Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.017986_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.017986Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:22:54.477 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-08T07:22:54.477 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-02-08T07:22:54.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:54.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:54.747 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:54.782 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:54.782 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:54.999 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:22:55.036 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612768974.739752_241.1","last_update":"2021-02-08T07:22:54.739752Z"}] 2021-02-08T07:22:55.037 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:22:55.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:55.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:22:55.037 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:22:55.077 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:22:55.077 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:22:55.299 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768909.778782_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:49.778782Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:22:55.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612768974.739752_241.1 target=1_1612768909.778782_219.1 2021-02-08T07:23:00.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:00.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:00.300 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:00.342 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:00.342 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:00.557 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768974.739752_241.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:54.739752Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:00.558 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:23:00.558 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:23:00.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:00.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:00.558 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:00.599 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:00.600 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:00.841 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768974.739752_241.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:54.739752Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:00.841 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:23:00.841 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:23:00.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:00.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:00.842 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:01.153 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768974.739752_241.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:54.739752Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:01.154 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:23:02.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:02.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:02.147 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:02.189 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:02.189 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:02.450 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:02.451 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:02.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:02.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:02.451 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:02.495 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:02.495 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:02.733 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:02.733 WARNING:rgw_multi.tests:bucket lhddgo-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000001.24.5 target= 2021-02-08T07:23:07.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:07.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:07.734 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:07.777 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:07.777 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:08.025 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:08.026 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:08.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:08.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:08.026 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:08.067 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:08.067 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:08.321 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:08.321 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:08.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:08.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:08.322 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:08.409 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:08.409 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:08.678 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:08.678 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:08.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:08.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:08.679 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:08.726 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:08.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:08.964 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:08.964 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:08.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:08.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:08.965 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:09.008 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:09.008 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:09.279 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:09.280 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:09.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:09.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:09.280 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:09.532 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:09.533 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:09.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:09.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:09.533 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:09.576 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:09.576 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:09.848 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:09.848 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:09.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:09.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:09.849 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:10.113 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:10.113 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:10.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:10.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:10.113 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:10.160 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:10.160 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:10.430 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:10.431 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:10.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:10.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:10.432 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:10.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:10.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:10.727 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:10.728 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:10.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:10.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:10.728 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:10.811 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:10.811 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:11.082 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:11.082 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:11.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:11.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:11.082 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:11.165 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:11.165 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:11.415 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:11.415 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:11.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:11.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:11.415 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:11.459 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:11.459 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:11.689 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:11.689 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:11.689 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-20 zones={test-zone2, test-zone1} 2021-02-08T07:23:12.602 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:23:12.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:12.612 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:23:12.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:12.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:12.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:12.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:12.855 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-20 zones={test-zone2, test-zone1} 2021-02-08T07:23:12.856 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-20 zones={test-zone3, test-zone1} 2021-02-08T07:23:13.773 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:23:13.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:13.823 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:23:13.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:13.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:13.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:13.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:14.027 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-20 zones={test-zone3, test-zone1} 2021-02-08T07:23:14.027 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-20 zones={test-zone3, test-zone2} 2021-02-08T07:23:14.637 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:23:20.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:20.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:23:21.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:21.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:21.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:21.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:21.370 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-20 zones={test-zone3, test-zone2} 2021-02-08T07:23:21.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:21.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:21.651 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:21.697 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:21.697 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:21.908 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:23:21.947 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612769001.644747_249.1","last_update":"2021-02-08T07:23:21.644747Z"}] 2021-02-08T07:23:21.947 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:23:21.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:21.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:21.948 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:21.998 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:21.998 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:22.226 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612768974.739752_241.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:54.739752Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:22.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769001.644747_249.1 target=1_1612768974.739752_241.1 2021-02-08T07:23:27.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:27.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:27.228 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:27.269 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:27.269 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:27.498 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769001.644747_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:23:21.644747Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:27.499 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:23:27.499 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:23:27.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:27.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:27.500 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:27.540 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:27.540 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:27.799 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769001.644747_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:23:21.644747Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:27.799 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:23:27.800 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:23:27.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:27.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:27.800 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:28.112 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769001.644747_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:23:21.644747Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:28.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:23:28.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:28.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:28.531 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:28.574 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:28.575 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:28.792 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.27.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:28.792 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:28.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:28.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:28.793 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:28.838 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:28.838 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:29.066 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:29.067 WARNING:rgw_multi.tests:bucket lhddgo-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.27.12 target=00000000001.24.5 2021-02-08T07:23:34.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:34.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:34.069 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:34.111 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:34.111 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:34.350 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.27.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:34.351 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:34.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:34.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:34.351 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:34.637 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.27.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:34.637 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:34.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:34.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:34.638 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:34.725 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:34.726 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:35.001 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.27.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:35.001 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:35.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:35.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:35.002 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:35.044 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:35.044 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:35.291 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.27.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:35.291 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:35.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:35.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:35.292 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:35.334 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:35.334 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:35.608 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.27.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:35.608 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:35.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:35.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:35.609 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:35.689 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:35.689 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:35.923 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.24.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:35.923 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:35.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:35.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:35.924 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:35.968 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:35.968 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:36.253 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:36.253 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:36.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:36.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:36.254 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:36.297 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:36.297 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:36.523 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.24.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:36.524 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:36.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:36.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:36.524 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:36.568 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:36.568 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:36.836 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:36.836 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:36.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:36.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:36.837 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:36.918 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:36.918 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:37.167 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.20.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:37.167 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:37.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:37.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:37.168 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:37.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:37.534 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:37.535 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:37.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:37.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:37.535 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:37.572 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:37.572 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:37.815 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.20.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:37.815 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:37.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:37.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:37.816 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:37.859 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:37.860 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:38.111 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:38.111 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:38.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-20 zones={test-zone2, test-zone1} 2021-02-08T07:23:39.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:23:39.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:39.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:39.502 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:23:39.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:39.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:39.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:39.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:39.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:39.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:39.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:39.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:39.888 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-20 zones={test-zone2, test-zone1} 2021-02-08T07:23:39.888 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-20 zones={test-zone3, test-zone1} 2021-02-08T07:23:40.396 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:23:40.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:40.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:40.451 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:23:40.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:40.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:40.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:40.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:40.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:40.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:40.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:40.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:40.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-20 zones={test-zone3, test-zone1} 2021-02-08T07:23:40.786 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-20 zones={test-zone3, test-zone2} 2021-02-08T07:23:40.797 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:23:40.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:40.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:40.843 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:23:40.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:40.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:23:40.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:40.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:40.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:41.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:41.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:41.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:23:41.197 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-20 zones={test-zone3, test-zone2} 2021-02-08T07:23:41.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:41.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:41.213 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:41.250 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:41.251 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:41.475 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:23:41.511 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612769021.207297_257.1","last_update":"2021-02-08T07:23:41.207297Z"}] 2021-02-08T07:23:41.512 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:23:41.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:41.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:41.512 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:41.554 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:41.554 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:41.782 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769001.644747_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:23:21.644747Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:41.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769021.207297_257.1 target=1_1612769001.644747_249.1 2021-02-08T07:23:46.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:48.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:48.664 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:48.706 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:48.706 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:48.927 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769021.207297_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:23:41.207297Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:48.928 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:23:48.928 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:23:48.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:48.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:48.928 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:48.971 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:48.971 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:49.235 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769021.207297_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:23:41.207297Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:49.235 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:23:49.235 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:23:49.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:49.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:49.236 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:49.545 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769021.207297_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:23:41.207297Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:23:49.546 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:23:50.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:50.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:50.319 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:50.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:50.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:50.607 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.31.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:50.608 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:50.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:50.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:50.608 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:50.654 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:50.654 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:50.900 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.27.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:50.900 WARNING:rgw_multi.tests:bucket lhddgo-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000005.31.17 target=00000000003.27.12 2021-02-08T07:23:55.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:55.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:55.902 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:55.945 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:55.946 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:56.195 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.31.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:56.195 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:56.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:56.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:56.196 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:56.237 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:56.488 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.31.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:56.489 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:56.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:56.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:56.489 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:56.573 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:56.573 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:56.825 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.31.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:56.826 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:56.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:56.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:56.826 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:56.865 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:56.865 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:57.110 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.31.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:57.110 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:57.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:57.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:57.111 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:57.158 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:57.158 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:57.430 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.31.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:57.430 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-20 2021-02-08T07:23:57.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:57.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:57.431 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:57.512 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:57.513 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:57.736 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.31.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:57.737 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:57.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:57.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:57.737 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:57.782 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:57.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:58.069 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.31.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:58.069 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:58.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:58.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:58.070 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:58.113 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:58.113 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:58.351 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.31.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:58.351 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:58.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:58.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:58.352 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:58.396 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:58.396 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:58.667 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.31.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:58.668 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-20 2021-02-08T07:23:58.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:58.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:58.668 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:58.752 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:58.752 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:59.015 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.27.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:59.015 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:59.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:59.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:59.015 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:59.105 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:59.105 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:59.383 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.27.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:59.384 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:59.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:59.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:59.384 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:59.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:59.429 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:59.678 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.27.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:23:59.679 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:59.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:59.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:23:59.679 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:23:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:23:59.723 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:23:59.971 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.27.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:23:59.971 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-20 2021-02-08T07:23:59.972 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-20 zones={test-zone2, test-zone1} 2021-02-08T07:24:01.362 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:01.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:01.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:01.369 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:01.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:01.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:01.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:01.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:01.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:01.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:01.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:01.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:01.768 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-20 zones={test-zone2, test-zone1} 2021-02-08T07:24:01.768 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-20 zones={test-zone3, test-zone1} 2021-02-08T07:24:01.938 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:01.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:01.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:01.991 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:01.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:01.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:02.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.379 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-20 zones={test-zone3, test-zone1} 2021-02-08T07:24:02.379 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-20 zones={test-zone3, test-zone2} 2021-02-08T07:24:02.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:02.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:02.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:02.443 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:02.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:02.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:02.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:02.784 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-20 zones={test-zone3, test-zone2} 2021-02-08T07:24:02.785 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-02-08T07:24:02.785 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-21 2021-02-08T07:24:02.820 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-22 2021-02-08T07:24:02.877 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-23 2021-02-08T07:24:02.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:02.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:02.990 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:03.026 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:03.026 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:03.243 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:24:03.282 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612769042.978250_281.1","last_update":"2021-02-08T07:24:02.978250Z"}] 2021-02-08T07:24:03.282 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:24:03.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:03.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:03.283 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:03.322 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:03.322 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:03.557 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769021.207297_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:23:41.207297Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:24:03.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769042.978250_281.1 target=1_1612769021.207297_257.1 2021-02-08T07:24:08.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:08.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:08.559 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:08.811 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769042.978250_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:24:02.978250Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:24:08.812 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:24:08.812 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:24:08.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:08.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:08.813 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:08.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:08.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:09.110 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769042.978250_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:24:02.978250Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:24:09.110 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:24:09.111 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:24:09.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:09.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:09.111 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:09.420 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769042.978250_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:24:02.978250Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:24:09.420 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:24:10.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:10.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:10.526 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:10.566 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:10.566 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:10.811 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.13.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:10.811 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-21 2021-02-08T07:24:10.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:10.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:10.858 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:10.859 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:11.074 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sDBRK9H0UYSaKTgBzpncneteH9lc-9G","ns":""},"count":1},"inc_marker":{"position":"00000000003.13.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:11.074 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-21 2021-02-08T07:24:11.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:11.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:11.075 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:11.115 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:11.359 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.13.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:11.359 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-21 2021-02-08T07:24:11.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:11.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:11.359 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:11.447 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:11.448 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:11.668 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sDBRK9H0UYSaKTgBzpncneteH9lc-9G","ns":""},"count":1},"inc_marker":{"position":"00000000003.13.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:11.669 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-21 2021-02-08T07:24:11.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:11.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:11.669 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:11.706 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:11.706 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:11.932 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.13.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:11.932 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-21 2021-02-08T07:24:11.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:11.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:11.933 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:11.977 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:11.977 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:12.245 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.13.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:12.246 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-21 2021-02-08T07:24:12.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:12.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:12.246 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:12.281 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:12.281 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:12.500 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:12.500 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-21 2021-02-08T07:24:12.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:12.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:12.501 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:12.545 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:12.545 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:12.804 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sDBRK9H0UYSaKTgBzpncneteH9lc-9G","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:12.804 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-21 2021-02-08T07:24:12.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:12.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:12.805 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:12.849 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:12.849 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:13.068 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:13.069 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-21 2021-02-08T07:24:13.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:13.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:13.069 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:13.110 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:13.110 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:13.356 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sDBRK9H0UYSaKTgBzpncneteH9lc-9G","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:13.357 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-21 2021-02-08T07:24:13.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:13.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:13.357 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:13.442 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:13.442 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:13.692 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:13.693 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-21 2021-02-08T07:24:13.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:13.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:13.693 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:13.775 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:13.775 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:14.055 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sDBRK9H0UYSaKTgBzpncneteH9lc-9G","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:14.055 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-21 2021-02-08T07:24:14.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:14.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:14.056 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:14.137 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:14.137 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:14.385 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:14.385 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-21 2021-02-08T07:24:14.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:14.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:14.386 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:14.657 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sDBRK9H0UYSaKTgBzpncneteH9lc-9G","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:14.658 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-21 2021-02-08T07:24:14.658 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-21 zones={test-zone2, test-zone1} 2021-02-08T07:24:15.532 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:15.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:15.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:15.539 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:15.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:15.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:15.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:15.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:15.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:15.738 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-21 zones={test-zone2, test-zone1} 2021-02-08T07:24:15.738 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-21 zones={test-zone3, test-zone1} 2021-02-08T07:24:16.521 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:16.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:16.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:16.567 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:16.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:16.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:16.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:16.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:16.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:16.719 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-21 zones={test-zone3, test-zone1} 2021-02-08T07:24:16.719 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-21 zones={test-zone3, test-zone2} 2021-02-08T07:24:16.729 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:16.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:16.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:16.779 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:16.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:16.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:16.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:16.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:16.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:16.929 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-21 zones={test-zone3, test-zone2} 2021-02-08T07:24:16.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:16.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:16.929 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:16.970 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:17.232 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:17.232 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-22 2021-02-08T07:24:17.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:17.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:17.233 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:17.277 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:17.277 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:17.534 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dXuUfVX0Ide7dJbGEPzlC4prRxLE5ou","ns":""},"count":2},"inc_marker":{"position":"00000000002.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:17.547 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-22 2021-02-08T07:24:17.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:17.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:17.548 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:17.591 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:17.592 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:17.848 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:17.848 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-22 2021-02-08T07:24:17.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:17.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:17.849 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:17.935 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:17.935 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:18.209 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dXuUfVX0Ide7dJbGEPzlC4prRxLE5ou","ns":""},"count":2},"inc_marker":{"position":"00000000002.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:18.209 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-22 2021-02-08T07:24:18.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:18.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:18.210 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:18.249 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:18.249 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:18.506 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:18.507 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-22 2021-02-08T07:24:18.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:18.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:18.507 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:18.820 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:18.821 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-22 2021-02-08T07:24:18.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:18.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:18.821 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:18.902 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:18.902 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:19.125 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:19.125 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-22 2021-02-08T07:24:19.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:19.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:19.125 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:19.170 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:19.170 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:19.449 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dXuUfVX0Ide7dJbGEPzlC4prRxLE5ou","ns":""},"count":2},"inc_marker":{"position":"00000000003.18.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:19.449 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-22 2021-02-08T07:24:19.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:19.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:19.450 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:19.708 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:19.708 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-22 2021-02-08T07:24:19.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:19.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:19.709 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:19.753 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:19.753 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:20.041 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dXuUfVX0Ide7dJbGEPzlC4prRxLE5ou","ns":""},"count":2},"inc_marker":{"position":"00000000003.18.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:20.042 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-22 2021-02-08T07:24:20.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:20.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:20.042 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:20.084 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:20.084 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:20.354 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.16.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:20.354 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-22 2021-02-08T07:24:20.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:20.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:20.355 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:20.440 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:20.440 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:20.709 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dXuUfVX0Ide7dJbGEPzlC4prRxLE5ou","ns":""},"count":2},"inc_marker":{"position":"00000000002.16.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:20.709 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-22 2021-02-08T07:24:20.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:20.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:20.710 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:20.792 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:20.792 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:21.036 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.16.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:21.037 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-22 2021-02-08T07:24:21.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:21.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:21.037 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:21.081 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:21.081 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:21.327 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dXuUfVX0Ide7dJbGEPzlC4prRxLE5ou","ns":""},"count":2},"inc_marker":{"position":"00000000002.16.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:21.327 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-22 2021-02-08T07:24:21.327 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-22 zones={test-zone2, test-zone1} 2021-02-08T07:24:21.779 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:21.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:21.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:21.786 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:21.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:21.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:21.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:21.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:21.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:21.986 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-22 zones={test-zone2, test-zone1} 2021-02-08T07:24:21.986 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-22 zones={test-zone3, test-zone1} 2021-02-08T07:24:22.634 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:22.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:22.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:22.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:22.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:22.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:22.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:22.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:22.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:22.921 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-22 zones={test-zone3, test-zone1} 2021-02-08T07:24:22.922 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-22 zones={test-zone3, test-zone2} 2021-02-08T07:24:22.932 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:22.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:22.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:22.983 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:22.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:22.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:23.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:23.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:23.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:23.161 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-22 zones={test-zone3, test-zone2} 2021-02-08T07:24:23.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:23.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:23.162 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:23.208 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:23.208 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:23.456 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:23.457 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-23 2021-02-08T07:24:23.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:23.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:23.457 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:23.502 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:23.502 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:23.745 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"CTFNkY9kzBclZINO9.U7i03DjapN.tq","ns":""},"count":2},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:23.746 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-23 2021-02-08T07:24:23.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:23.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:23.747 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:23.794 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:23.794 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:24.251 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:24.251 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-23 2021-02-08T07:24:24.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:24.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:24.252 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:24.337 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:24.593 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"CTFNkY9kzBclZINO9.U7i03DjapN.tq","ns":""},"count":2},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:24.594 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-23 2021-02-08T07:24:24.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:24.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:24.594 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:24.629 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:24.629 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:24.864 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:24.864 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-23 2021-02-08T07:24:24.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:24.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:24.864 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:24.908 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:24.908 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:25.183 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:25.183 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-23 2021-02-08T07:24:25.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:25.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:25.184 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:25.265 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:25.265 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:25.496 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.19.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:25.496 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-23 2021-02-08T07:24:25.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:25.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:25.497 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:25.547 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:25.547 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:25.820 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"CTFNkY9kzBclZINO9.U7i03DjapN.tq","ns":""},"count":2},"inc_marker":{"position":"00000000002.19.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:25.821 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-23 2021-02-08T07:24:25.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:25.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:25.821 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:25.864 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:25.864 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:26.091 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.19.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:26.091 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-23 2021-02-08T07:24:26.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:26.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:26.091 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:26.135 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:26.135 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:26.409 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"CTFNkY9kzBclZINO9.U7i03DjapN.tq","ns":""},"count":2},"inc_marker":{"position":"00000000002.19.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:26.409 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-23 2021-02-08T07:24:26.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:26.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:26.410 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:26.493 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:26.494 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:26.747 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.25.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:26.747 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-23 2021-02-08T07:24:26.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:26.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:26.748 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:26.831 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:26.831 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:27.110 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"CTFNkY9kzBclZINO9.U7i03DjapN.tq","ns":""},"count":2},"inc_marker":{"position":"00000000003.25.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:27.110 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-23 2021-02-08T07:24:27.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:27.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:27.111 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:27.194 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:27.442 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.25.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:27.442 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-23 2021-02-08T07:24:27.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:27.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:27.443 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:27.486 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:27.486 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:27.737 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"CTFNkY9kzBclZINO9.U7i03DjapN.tq","ns":""},"count":2},"inc_marker":{"position":"00000000003.25.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:27.738 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-23 2021-02-08T07:24:27.738 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-23 zones={test-zone2, test-zone1} 2021-02-08T07:24:28.847 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:28.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:28.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:28.855 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:28.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:28.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:28.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:28.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:28.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:29.054 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-23 zones={test-zone2, test-zone1} 2021-02-08T07:24:29.054 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-23 zones={test-zone3, test-zone1} 2021-02-08T07:24:29.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:29.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:29.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:29.195 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:29.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:29.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:29.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:29.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:29.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:29.394 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-23 zones={test-zone3, test-zone1} 2021-02-08T07:24:29.395 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-23 zones={test-zone3, test-zone2} 2021-02-08T07:24:29.405 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:29.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:29.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:29.451 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:29.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:29.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:29.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:29.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:29.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:29.606 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-23 zones={test-zone3, test-zone2} 2021-02-08T07:24:29.606 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-02-08T07:24:29.607 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-24 2021-02-08T07:24:29.643 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-25 2021-02-08T07:24:29.700 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-26 2021-02-08T07:24:29.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:29.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:29.867 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:29.904 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:29.904 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:30.136 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:24:30.175 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612769069.857413_321.1","last_update":"2021-02-08T07:24:29.857413Z"}] 2021-02-08T07:24:30.176 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:24:30.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:30.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:30.176 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:30.217 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:30.217 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:30.440 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769042.978250_281.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:24:02.978250Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:24:30.441 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769069.857413_321.1 target=1_1612769042.978250_281.1 2021-02-08T07:24:35.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:35.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:35.442 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:35.483 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:35.484 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:35.710 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769069.857413_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:24:29.857413Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:24:35.711 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:24:35.711 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:24:35.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:35.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:35.712 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:35.751 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:35.751 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:36.003 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769069.857413_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:24:29.857413Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:24:36.004 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:24:36.004 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:24:36.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:36.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:36.004 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:36.042 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:36.043 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:36.318 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769069.857413_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:24:29.857413Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:24:36.318 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:24:37.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:37.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:37.277 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:37.318 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:37.318 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:37.575 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.14.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:37.576 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-24 2021-02-08T07:24:37.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:37.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:37.576 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:37.623 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:37.623 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:37.838 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:37.838 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-24 2021-02-08T07:24:37.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:37.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:37.839 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:37.879 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:37.880 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:38.148 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.14.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:38.148 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-24 2021-02-08T07:24:38.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:38.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:38.148 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:38.231 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:38.232 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:38.476 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:38.476 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-24 2021-02-08T07:24:38.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:38.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:38.477 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:38.558 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:38.559 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:38.805 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.14.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:38.805 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-24 2021-02-08T07:24:38.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:38.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:38.806 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:38.849 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:38.849 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:39.125 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.14.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:39.125 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-24 2021-02-08T07:24:39.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:39.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:39.126 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:39.209 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:39.209 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:39.417 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.9.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:39.417 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-24 2021-02-08T07:24:39.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:39.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:39.418 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:39.464 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:39.464 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:39.700 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:39.701 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-24 2021-02-08T07:24:39.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:39.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:39.701 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:39.744 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:39.744 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:39.974 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.9.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:39.975 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-24 2021-02-08T07:24:39.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:39.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:39.976 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:40.019 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:40.019 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:40.291 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:40.291 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-24 2021-02-08T07:24:40.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:40.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:40.292 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:40.376 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:40.377 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:40.653 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.7.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:40.653 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-24 2021-02-08T07:24:40.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:40.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:40.654 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:40.740 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:41.029 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:41.029 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-24 2021-02-08T07:24:41.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:41.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:41.030 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:41.113 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:41.113 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:41.368 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.7.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:41.368 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-24 2021-02-08T07:24:41.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:41.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:41.369 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:41.415 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:41.415 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:41.660 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:41.661 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-24 2021-02-08T07:24:41.661 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-24 zones={test-zone2, test-zone1} 2021-02-08T07:24:42.193 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:42.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:42.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:42.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:42.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:42.308 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-24 zones={test-zone2, test-zone1} 2021-02-08T07:24:42.308 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-24 zones={test-zone3, test-zone1} 2021-02-08T07:24:42.765 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:42.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:42.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:42.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:42.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:42.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-24 zones={test-zone3, test-zone1} 2021-02-08T07:24:42.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-24 zones={test-zone3, test-zone2} 2021-02-08T07:24:42.905 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:42.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:42.951 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:42.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:43.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:43.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-24 zones={test-zone3, test-zone2} 2021-02-08T07:24:43.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:43.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:43.019 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:43.059 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:43.060 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:43.339 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.152.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:43.340 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-25 2021-02-08T07:24:43.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:43.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:43.340 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:43.384 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:43.384 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:43.627 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.152.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:43.627 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-25 2021-02-08T07:24:43.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:43.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:43.628 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:43.682 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:43.682 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:43.931 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.152.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:43.932 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-25 2021-02-08T07:24:43.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:43.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:43.933 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:44.031 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:44.031 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:44.314 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.152.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:44.314 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-25 2021-02-08T07:24:44.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:44.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:44.315 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:44.352 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:44.625 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.152.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:44.626 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-25 2021-02-08T07:24:44.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:44.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:44.626 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:44.671 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:44.943 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.152.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:44.943 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-25 2021-02-08T07:24:44.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:44.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:44.944 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:45.024 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:45.024 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:45.242 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.201.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:45.242 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-25 2021-02-08T07:24:45.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:45.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:45.243 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:45.286 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:45.287 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:45.576 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.201.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:45.576 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-25 2021-02-08T07:24:45.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:45.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:45.577 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:45.618 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:45.618 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:45.843 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.201.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:45.843 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-25 2021-02-08T07:24:45.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:45.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:45.844 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:45.887 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:45.887 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:46.153 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.201.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:46.154 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-25 2021-02-08T07:24:46.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:46.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:46.154 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:46.239 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:46.239 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:46.500 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.159.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:46.500 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-25 2021-02-08T07:24:46.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:46.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:46.501 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:46.583 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:46.584 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:46.858 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.159.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:46.859 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-25 2021-02-08T07:24:46.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:46.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:46.859 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:46.941 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:46.941 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:47.206 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.159.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:47.206 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-25 2021-02-08T07:24:47.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:47.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:47.207 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:47.250 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:47.250 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:47.502 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.159.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:47.503 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-25 2021-02-08T07:24:47.503 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-25 zones={test-zone2, test-zone1} 2021-02-08T07:24:48.559 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:48.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:48.567 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:48.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:48.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:48.686 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-25 zones={test-zone2, test-zone1} 2021-02-08T07:24:48.686 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-25 zones={test-zone3, test-zone1} 2021-02-08T07:24:48.909 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:52.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:52.154 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:52.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:52.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:52.347 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-25 zones={test-zone3, test-zone1} 2021-02-08T07:24:52.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-25 zones={test-zone3, test-zone2} 2021-02-08T07:24:52.898 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:52.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:52.908 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:52.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:53.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:53.029 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-25 zones={test-zone3, test-zone2} 2021-02-08T07:24:53.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:53.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:53.029 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:53.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:53.070 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:53.329 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:53.329 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-26 2021-02-08T07:24:53.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:53.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:53.330 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:53.373 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:53.373 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:53.625 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:53.626 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-26 2021-02-08T07:24:53.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:53.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:53.626 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:53.666 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:53.667 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:53.917 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:53.917 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-26 2021-02-08T07:24:53.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:53.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:53.918 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:54.010 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:54.010 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:54.288 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:54.289 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-26 2021-02-08T07:24:54.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:54.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:54.289 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:54.333 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:54.334 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:54.583 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:54.584 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-26 2021-02-08T07:24:54.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:54.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:54.584 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:54.631 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:54.631 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:54.890 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.16.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:54.891 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-26 2021-02-08T07:24:54.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:54.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:54.891 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:54.972 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:54.973 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:55.204 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.12.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:55.205 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-26 2021-02-08T07:24:55.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:55.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:55.206 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:55.252 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:55.252 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:55.618 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:55.619 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-26 2021-02-08T07:24:55.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:55.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:55.619 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:55.891 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.12.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:55.891 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-26 2021-02-08T07:24:55.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:55.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:55.891 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:55.937 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:55.937 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:56.214 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:56.215 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-26 2021-02-08T07:24:56.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:56.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:56.215 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:56.254 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:56.521 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.21.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:56.521 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-26 2021-02-08T07:24:56.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:56.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:56.522 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:56.609 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:56.609 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:56.895 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:56.896 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-26 2021-02-08T07:24:56.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:56.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:56.896 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:56.938 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:57.192 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.21.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:24:57.192 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-26 2021-02-08T07:24:57.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:57.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:24:57.193 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:24:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:24:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:24:57.492 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:24:57.492 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-26 2021-02-08T07:24:57.492 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-26 zones={test-zone2, test-zone1} 2021-02-08T07:24:58.497 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:58.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:58.504 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:58.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:58.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:58.615 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-26 zones={test-zone2, test-zone1} 2021-02-08T07:24:58.615 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-26 zones={test-zone3, test-zone1} 2021-02-08T07:24:58.651 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:58.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:58.699 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:58.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:58.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:58.774 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-26 zones={test-zone3, test-zone1} 2021-02-08T07:24:58.774 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-26 zones={test-zone3, test-zone2} 2021-02-08T07:24:58.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:24:58.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:58.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:24:58.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-08T07:24:58.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-08T07:24:58.902 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-26 zones={test-zone3, test-zone2} 2021-02-08T07:24:58.902 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-02-08T07:24:58.903 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-27 2021-02-08T07:24:58.940 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-28 2021-02-08T07:24:59.002 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-29 2021-02-08T07:24:59.262 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2021-02-08T07:24:59.263 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-30 2021-02-08T07:24:59.303 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-31 2021-02-08T07:24:59.363 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-32 2021-02-08T07:24:59.607 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2021-02-08T07:24:59.607 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-33 2021-02-08T07:24:59.648 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-34 2021-02-08T07:24:59.713 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-35 2021-02-08T07:25:02.219 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2021-02-08T07:25:02.219 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-36 2021-02-08T07:25:02.782 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-37 2021-02-08T07:25:03.158 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-38 2021-02-08T07:25:03.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:03.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:03.223 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:25:03.260 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:03.261 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:03.456 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:25:03.494 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612769103.194698_403.1","last_update":"2021-02-08T07:25:03.194698Z"}] 2021-02-08T07:25:03.495 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:25:03.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:03.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:03.495 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:25:03.537 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:03.538 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:03.742 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769103.194698_403.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:03.194698Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:25:03.742 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:25:03.742 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:25:03.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:03.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:03.743 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:25:03.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:03.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:04.031 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769103.194698_403.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:03.194698Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:25:04.031 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:25:04.031 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:25:04.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:04.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:04.032 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:25:04.070 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:04.070 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:04.348 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769103.194698_403.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:03.194698Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:25:04.349 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:25:06.679 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2021-02-08T07:25:06.679 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-39 2021-02-08T07:25:06.712 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-40 2021-02-08T07:25:07.076 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-41 2021-02-08T07:25:07.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:07.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:07.137 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:25:07.170 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:07.171 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:07.375 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=d4b11cd5-1a7c-4383-afcf-402844a1791d 2021-02-08T07:25:07.409 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612768579.532632_32.1","last_update":"2021-02-08T07:16:19.532632Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612768739.752484_117.1","last_update":"2021-02-08T07:18:59.752484Z"},{"marker":"1_1612769107.111803_416.1","last_update":"2021-02-08T07:25:07.111803Z"}] 2021-02-08T07:25:07.410 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:25:07.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:07.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:07.411 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:25:07.445 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:07.445 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:07.664 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769103.194698_403.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:03.194698Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:25:07.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769107.111803_416.1 target=1_1612769103.194698_403.1 2021-02-08T07:25:12.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:12.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:12.666 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:25:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:12.937 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769107.111803_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.111803Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:25:12.938 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:25:12.938 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:25:12.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:12.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:12.938 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:25:12.978 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:12.979 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:13.224 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769107.111803_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.111803Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:25:13.225 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:25:13.225 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:25:13.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:13.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:25:13.225 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:25:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:13.639 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769107.111803_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.111803Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:25:13.640 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:25:13.640 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-08T07:25:13.640 INFO:teuthology.orchestra.run:waiting for 300 2021-02-08T07:25:13.641 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:25:13.637+0000 7fbdf0f48700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 16911) UID: 0 2021-02-08T07:25:13.641 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:25:13.637+0000 7fbe0860a0c0 -1 shutting down 2021-02-08T07:25:15.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-08T07:25:19.742 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-08T07:25:19.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:25:19.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:25:19.742 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:25:19.781 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:19.781 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:19.808 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:25:19.797+0000 7f247d22c080 0 NOTICE: overriding master zone: 757db88c-ee56-405d-bd0c-79b9f5b05722 2021-02-08T07:25:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:25:19.801+0000 7f247d22c080 0 failed reading obj info from .rgw.root:zone_info.757db88c-ee56-405d-bd0c-79b9f5b05722: (2) No such file or directory 2021-02-08T07:25:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:25:19.801+0000 7f247d22c080 0 WARNING: could not read zone params for zone id=757db88c-ee56-405d-bd0c-79b9f5b05722 name=test-zone1 2021-02-08T07:25:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:25:19.801+0000 7f247d22c080 0 failed reading obj info from .rgw.root:zone_info.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d: (2) No such file or directory 2021-02-08T07:25:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:25:19.801+0000 7f247d22c080 0 WARNING: could not read zone params for zone id=8e9c5146-39d0-40f8-b41d-7c27e66f6e3d name=test-zone3 2021-02-08T07:25:19.823 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:25:19.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:25:19.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:25:19.824 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:25:19.903 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:25:19.903 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:25:20.210 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","epoch":1,"predecessor_uuid":"d4b11cd5-1a7c-4383-afcf-402844a1791d","sync_status":["1_1612768579.532632_32.1","","1_1612768739.752484_117.1","1_1612769107.111803_416.1"],"period_map":{"id":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"c897876e-5513-44a9-a99f-ee5993968e59","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"c897876e-5513-44a9-a99f-ee5993968e59","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":3} 2021-02-08T07:25:20.210 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2021-02-08T07:25:21.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-08T07:26:20.211 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-42 2021-02-08T07:26:20.290 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-43 2021-02-08T07:26:20.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:26:20.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:26:20.327 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:26:20.361 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:26:20.361 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:26:20.557 INFO:teuthology.orchestra.run.smithi110.stderr:No --period given, using current period=49be1c29-d1bd-4210-9e84-e35ba7fb03cf 2021-02-08T07:26:20.590 INFO:teuthology.orchestra.run.smithi110.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612769180.317275_302.1","last_update":"2021-02-08T07:26:20.317275Z"}] 2021-02-08T07:26:20.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2021-02-08T07:26:20.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:26:20.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:26:20.591 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:26:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:26:20.632 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:26:20.887 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-02-08T07:26:20.887 WARNING:rgw_multi.tests:zone test-zone1 behind master: shard=3 master=1_1612769180.317275_302.1 target= 2021-02-08T07:26:25.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:26:25.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:26:25.888 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:26:25.930 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:26:25.930 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:26:26.187 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"1_1612769180.317275_302.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:26:20.317275Z","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-02-08T07:26:26.188 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2021-02-08T07:26:26.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:26:26.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:26:26.189 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:26:26.267 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:26:26.267 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:26:26.297 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:26:26.288+0000 7f68a2068080 0 NOTICE: overriding master zone: c897876e-5513-44a9-a99f-ee5993968e59 2021-02-08T07:26:26.297 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:26:26.292+0000 7f68a2068080 0 failed reading obj info from .rgw.root:zone_info.4c93366d-9954-4fa2-95e3-91873db27ab5: (2) No such file or directory 2021-02-08T07:26:26.298 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:26:26.292+0000 7f68a2068080 0 WARNING: could not read zone params for zone id=4c93366d-9954-4fa2-95e3-91873db27ab5 name=test-zone4 2021-02-08T07:26:26.300 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:26:26.292+0000 7f68a2068080 0 failed reading obj info from .rgw.root:zone_info.c897876e-5513-44a9-a99f-ee5993968e59: (2) No such file or directory 2021-02-08T07:26:26.300 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:26:26.292+0000 7f68a2068080 0 WARNING: could not read zone params for zone id=c897876e-5513-44a9-a99f-ee5993968e59 name=test-zone2 2021-02-08T07:26:26.314 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:26:26.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:26:26.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:26:26.315 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:26:26.396 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:26:26.396 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:26:26.715 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","epoch":1,"predecessor_uuid":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","sync_status":["","","","1_1612769180.317275_302.1"],"period_map":{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":4} 2021-02-08T07:26:26.716 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2021-02-08T07:27:26.717 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-44 2021-02-08T07:27:26.762 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-45 2021-02-08T07:27:26.837 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-08T07:27:26.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-08T07:27:26.840 INFO:tasks.rgw.c1.client.1:Started 2021-02-08T07:27:26.840 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2021-02-08T07:27:26.855 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:27:26.856 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:27:26.859 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2021-02-08T07:27:26.860 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-08T07:27:27.860 DEBUG:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2021-02-08T07:27:27.876 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-08T07:27:27.876 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2021-02-08T07:27:27.881 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-02-08T07:27:27.882 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2021-02-08T07:27:27.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:27.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:27.882 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:27.967 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:27.967 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:28.191 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=38375dfb-4fb1-40a7-81e1-7d3f26b8d208 2021-02-08T07:27:28.221 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612769246.810829_239.1","last_update":"2021-02-08T07:27:26.810829Z"}] 2021-02-08T07:27:28.221 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:27:28.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:28.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:28.222 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:28.260 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:28.260 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:28.486 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"d4b11cd5-1a7c-4383-afcf-402844a1791d","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769107.111803_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.111803Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:27:28.487 WARNING:rgw_multi.tests:zone test-zone2 is syncing realm epoch=2, behind current realm epoch=4 2021-02-08T07:27:33.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:33.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:33.489 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:33.528 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:33.528 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:33.759 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769246.810829_239.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:26.810829Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:27:33.759 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:27:33.759 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:27:33.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:33.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:33.760 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:33.800 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:33.800 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:34.049 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769246.810829_239.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:26.810829Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:27:34.049 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:27:34.049 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:27:34.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:34.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:34.050 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:34.090 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:34.090 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:34.362 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769246.810829_239.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:26.810829Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:27:34.362 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:27:34.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-39 zones={test-zone2, test-zone1} 2021-02-08T07:27:35.363 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:35.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:35.479 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-39 zones={test-zone2, test-zone1} 2021-02-08T07:27:35.479 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-39 zones={test-zone3, test-zone1} 2021-02-08T07:27:35.499 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:35.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:35.620 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-39 zones={test-zone3, test-zone1} 2021-02-08T07:27:35.620 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-39 zones={test-zone1, test-zone2} 2021-02-08T07:27:35.629 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:35.683 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:35.745 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-39 zones={test-zone1, test-zone2} 2021-02-08T07:27:35.745 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-39 zones={test-zone3, test-zone2} 2021-02-08T07:27:35.756 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:35.807 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:35.869 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-39 zones={test-zone3, test-zone2} 2021-02-08T07:27:35.869 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-39 zones={test-zone1, test-zone3} 2021-02-08T07:27:35.880 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:35.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:35.989 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-39 zones={test-zone1, test-zone3} 2021-02-08T07:27:35.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-39 zones={test-zone2, test-zone3} 2021-02-08T07:27:36.000 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:36.055 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:36.123 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-39 zones={test-zone2, test-zone3} 2021-02-08T07:27:36.123 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-40 zones={test-zone2, test-zone1} 2021-02-08T07:27:36.136 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:36.182 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:36.255 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-40 zones={test-zone2, test-zone1} 2021-02-08T07:27:36.255 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-40 zones={test-zone3, test-zone1} 2021-02-08T07:27:36.266 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:36.318 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:36.382 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-40 zones={test-zone3, test-zone1} 2021-02-08T07:27:36.382 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-40 zones={test-zone1, test-zone2} 2021-02-08T07:27:36.394 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:36.442 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:36.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-40 zones={test-zone1, test-zone2} 2021-02-08T07:27:36.510 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-40 zones={test-zone3, test-zone2} 2021-02-08T07:27:36.522 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:36.571 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:36.638 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-40 zones={test-zone3, test-zone2} 2021-02-08T07:27:36.639 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-40 zones={test-zone1, test-zone3} 2021-02-08T07:27:36.653 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:36.699 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:36.765 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-40 zones={test-zone1, test-zone3} 2021-02-08T07:27:36.765 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-40 zones={test-zone2, test-zone3} 2021-02-08T07:27:36.775 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:36.827 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:36.891 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-40 zones={test-zone2, test-zone3} 2021-02-08T07:27:36.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-41 zones={test-zone2, test-zone1} 2021-02-08T07:27:36.904 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:36.951 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:37.024 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-41 zones={test-zone2, test-zone1} 2021-02-08T07:27:37.024 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-41 zones={test-zone3, test-zone1} 2021-02-08T07:27:37.036 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:37.082 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:37.147 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-41 zones={test-zone3, test-zone1} 2021-02-08T07:27:37.147 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-41 zones={test-zone1, test-zone2} 2021-02-08T07:27:37.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:37.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:37.272 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-41 zones={test-zone1, test-zone2} 2021-02-08T07:27:37.273 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-41 zones={test-zone3, test-zone2} 2021-02-08T07:27:37.284 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:37.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:37.398 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-41 zones={test-zone3, test-zone2} 2021-02-08T07:27:37.398 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-41 zones={test-zone1, test-zone3} 2021-02-08T07:27:37.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:37.458 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:37.523 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-41 zones={test-zone1, test-zone3} 2021-02-08T07:27:37.523 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-41 zones={test-zone2, test-zone3} 2021-02-08T07:27:37.532 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:37.578 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:37.644 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-41 zones={test-zone2, test-zone3} 2021-02-08T07:27:37.645 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-42 zones={test-zone2, test-zone1} 2021-02-08T07:27:37.654 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:37.702 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:37.769 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-42 zones={test-zone2, test-zone1} 2021-02-08T07:27:37.769 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-42 zones={test-zone3, test-zone1} 2021-02-08T07:27:37.779 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:37.830 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:37.898 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-42 zones={test-zone3, test-zone1} 2021-02-08T07:27:37.898 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-42 zones={test-zone1, test-zone2} 2021-02-08T07:27:37.909 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:37.955 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:38.024 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-42 zones={test-zone1, test-zone2} 2021-02-08T07:27:38.024 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-42 zones={test-zone3, test-zone2} 2021-02-08T07:27:38.037 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:38.090 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:38.157 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-42 zones={test-zone3, test-zone2} 2021-02-08T07:27:38.157 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-42 zones={test-zone1, test-zone3} 2021-02-08T07:27:38.169 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:38.222 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:38.291 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-42 zones={test-zone1, test-zone3} 2021-02-08T07:27:38.292 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-42 zones={test-zone2, test-zone3} 2021-02-08T07:27:38.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:38.351 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:38.422 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-42 zones={test-zone2, test-zone3} 2021-02-08T07:27:38.422 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-43 zones={test-zone2, test-zone1} 2021-02-08T07:27:38.432 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:38.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:38.547 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-43 zones={test-zone2, test-zone1} 2021-02-08T07:27:38.547 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-43 zones={test-zone3, test-zone1} 2021-02-08T07:27:38.556 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:38.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:38.670 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-43 zones={test-zone3, test-zone1} 2021-02-08T07:27:38.671 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-43 zones={test-zone1, test-zone2} 2021-02-08T07:27:38.683 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:38.730 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:38.802 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-43 zones={test-zone1, test-zone2} 2021-02-08T07:27:38.802 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-43 zones={test-zone3, test-zone2} 2021-02-08T07:27:38.813 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:38.867 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:38.929 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-43 zones={test-zone3, test-zone2} 2021-02-08T07:27:38.929 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-43 zones={test-zone1, test-zone3} 2021-02-08T07:27:38.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:38.990 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:39.053 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-43 zones={test-zone1, test-zone3} 2021-02-08T07:27:39.054 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-43 zones={test-zone2, test-zone3} 2021-02-08T07:27:39.064 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:39.114 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:39.179 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-43 zones={test-zone2, test-zone3} 2021-02-08T07:27:39.179 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-44 zones={test-zone2, test-zone1} 2021-02-08T07:27:39.190 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:39.239 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:39.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-44 zones={test-zone2, test-zone1} 2021-02-08T07:27:39.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-44 zones={test-zone3, test-zone1} 2021-02-08T07:27:39.312 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:39.358 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:39.427 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-44 zones={test-zone3, test-zone1} 2021-02-08T07:27:39.427 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-44 zones={test-zone1, test-zone2} 2021-02-08T07:27:39.438 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:39.483 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:39.544 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-44 zones={test-zone1, test-zone2} 2021-02-08T07:27:39.545 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-44 zones={test-zone3, test-zone2} 2021-02-08T07:27:39.554 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:39.606 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:39.674 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-44 zones={test-zone3, test-zone2} 2021-02-08T07:27:39.674 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-44 zones={test-zone1, test-zone3} 2021-02-08T07:27:39.685 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:39.735 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:39.800 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-44 zones={test-zone1, test-zone3} 2021-02-08T07:27:39.800 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-44 zones={test-zone2, test-zone3} 2021-02-08T07:27:39.811 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:39.859 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:39.923 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-44 zones={test-zone2, test-zone3} 2021-02-08T07:27:39.923 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-45 zones={test-zone2, test-zone1} 2021-02-08T07:27:39.934 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:39.987 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:40.051 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-45 zones={test-zone2, test-zone1} 2021-02-08T07:27:40.051 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-45 zones={test-zone3, test-zone1} 2021-02-08T07:27:40.061 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:40.107 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:40.170 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-45 zones={test-zone3, test-zone1} 2021-02-08T07:27:40.170 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-45 zones={test-zone1, test-zone2} 2021-02-08T07:27:40.183 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:40.230 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:40.297 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-45 zones={test-zone1, test-zone2} 2021-02-08T07:27:40.297 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-45 zones={test-zone3, test-zone2} 2021-02-08T07:27:40.309 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:40.363 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:40.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-45 zones={test-zone3, test-zone2} 2021-02-08T07:27:40.425 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-45 zones={test-zone1, test-zone3} 2021-02-08T07:27:40.437 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:40.483 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:40.551 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-45 zones={test-zone1, test-zone3} 2021-02-08T07:27:40.551 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-45 zones={test-zone2, test-zone3} 2021-02-08T07:27:40.561 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:27:40.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:27:40.671 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-45 zones={test-zone2, test-zone3} 2021-02-08T07:27:40.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:40.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:40.672 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period d4b11cd5-1a7c-4383-afcf-402844a1791d --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:40.708 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:40.982 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1612768579.522956_30.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-08T07:16:19.522956903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GTmvG-EF-9cF89V4myGNcR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768579.532632_32.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-08T07:16:19.532632715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GTmvG-EF-9cF89V4myGNcR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.649968_40.1","section":"bucket.instance","name":"lhddgo-1:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1","timestamp":"2021-02-08T07:18:21.649968470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e2Im8Aginrg7vTQ5Uq0UExs","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.675446_41.1","section":"bucket.instance","name":"lhddgo-1:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1","timestamp":"2021-02-08T07:18:21.675446655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e2Im8Aginrg7vTQ5Uq0UExs","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.677384_42.1","section":"bucket","name":"lhddgo-1","timestamp":"2021-02-08T07:18:21.677384640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m_Fn4BKOiM5g7mlQklEvF8h","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.681366_43.1","section":"bucket","name":"lhddgo-1","timestamp":"2021-02-08T07:18:21.681366592Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m_Fn4BKOiM5g7mlQklEvF8h","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.702503_44.1","section":"bucket.instance","name":"lhddgo-2:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2","timestamp":"2021-02-08T07:18:21.702503042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6qX65q9Zll4vnVd5kG0CkcV","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.708160_45.1","section":"bucket.instance","name":"lhddgo-2:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2","timestamp":"2021-02-08T07:18:21.708160313Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6qX65q9Zll4vnVd5kG0CkcV","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.710147_46.1","section":"bucket","name":"lhddgo-2","timestamp":"2021-02-08T07:18:21.710147097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1DMS1mNFwKdoHY2cnmaj9tt","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.714859_47.1","section":"bucket","name":"lhddgo-2","timestamp":"2021-02-08T07:18:21.714859854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1DMS1mNFwKdoHY2cnmaj9tt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768704.880852_48.1","section":"bucket.instance","name":"lhddgo-3:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3","timestamp":"2021-02-08T07:18:24.880852494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rOLGVtTLt3ajM6sppBkkd3e","ver":1},"status":{"status":"write"}}},{"id":"1_1612768704.886356_49.1","section":"bucket.instance","name":"lhddgo-3:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3","timestamp":"2021-02-08T07:18:24.886356317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rOLGVtTLt3ajM6sppBkkd3e","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768704.888224_50.1","section":"bucket","name":"lhddgo-3","timestamp":"2021-02-08T07:18:24.888224742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqx6QY3WLExCkHpK84fAXxr","ver":1},"status":{"status":"write"}}},{"id":"1_1612768704.893108_51.1","section":"bucket","name":"lhddgo-3","timestamp":"2021-02-08T07:18:24.893108028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqx6QY3WLExCkHpK84fAXxr","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.777993_52.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4","timestamp":"2021-02-08T07:18:32.777993072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BoYtQApnX7_2AtGejxpYaZO","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.783478_53.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4","timestamp":"2021-02-08T07:18:32.783478935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BoYtQApnX7_2AtGejxpYaZO","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.785319_54.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:32.785319651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fp7ohDPD_24JfOiBzo-rYXf","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.791923_55.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:32.791923400Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fp7ohDPD_24JfOiBzo-rYXf","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.814845_56.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5","timestamp":"2021-02-08T07:18:32.814845373Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4CPxPuDJA5f6reOzSejIeq","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.820494_57.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5","timestamp":"2021-02-08T07:18:32.820494612Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4CPxPuDJA5f6reOzSejIeq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.822303_58.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:32.822303238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_yHBtAEVD0Gyl9lvsZLfTL","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.830478_59.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:32.830478754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_yHBtAEVD0Gyl9lvsZLfTL","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.875274_60.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6","timestamp":"2021-02-08T07:18:32.875274066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIAw4BnGrFg8-fkZKIn5AkT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.883030_61.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6","timestamp":"2021-02-08T07:18:32.883030645Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIAw4BnGrFg8-fkZKIn5AkT","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.885168_62.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:32.885168872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hS--GkAzPDMbDD0Ah6ChxD9","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.891808_63.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:32.891808199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hS--GkAzPDMbDD0Ah6ChxD9","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.455112_64.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.7","timestamp":"2021-02-08T07:18:40.455112453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D64Kh1AQ6YPfhYh6ce23ln9","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.461143_65.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.7","timestamp":"2021-02-08T07:18:40.461143164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D64Kh1AQ6YPfhYh6ce23ln9","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.463040_66.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.463040385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1HoTcP5PU2Kgj55NSIKnwiu","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.467993_67.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.467993630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1HoTcP5PU2Kgj55NSIKnwiu","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.499332_68.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.8","timestamp":"2021-02-08T07:18:40.499332959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ghr6TovGAVCrhaoBiwCDcI0","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.504921_69.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.8","timestamp":"2021-02-08T07:18:40.504921848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ghr6TovGAVCrhaoBiwCDcI0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.507224_70.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.507224631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRWiK6cMflxrN6xErwMtL8C","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.511355_71.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.511355956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRWiK6cMflxrN6xErwMtL8C","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.543636_72.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.9","timestamp":"2021-02-08T07:18:40.543636213Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xlBzXd-b99YpCwUbXyq6T_I","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.549664_73.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.9","timestamp":"2021-02-08T07:18:40.549664095Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xlBzXd-b99YpCwUbXyq6T_I","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.551977_74.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.551977841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2tIbTzLnZNEgqsoMi4Kyu4g","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.556906_75.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.556906173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2tIbTzLnZNEgqsoMi4Kyu4g","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.598520_76.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.10","timestamp":"2021-02-08T07:18:40.598520764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_erUTYNTZcG6PubXCSk30xX6","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.603500_77.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.10","timestamp":"2021-02-08T07:18:40.603500331Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_erUTYNTZcG6PubXCSk30xX6","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.605348_78.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.605348512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3KtUz4rocFNL2_vS5Rfub48","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.608562_79.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.608562308Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3KtUz4rocFNL2_vS5Rfub48","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.654740_80.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.11","timestamp":"2021-02-08T07:18:40.654740480Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HT1xmRKztbydlE5i670276O","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.660209_81.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.11","timestamp":"2021-02-08T07:18:40.660209900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HT1xmRKztbydlE5i670276O","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.662252_82.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.662252850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7sCGVKD50Piys06LRi5mRnA","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.667287_83.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.667287114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7sCGVKD50Piys06LRi5mRnA","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.764760_84.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.12","timestamp":"2021-02-08T07:18:40.764760372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vhvVK_HjSPWSrSoiQ3B1sWk","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.770576_85.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.12","timestamp":"2021-02-08T07:18:40.770576911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vhvVK_HjSPWSrSoiQ3B1sWk","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.772523_86.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.772523598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2kLGuYGaCkV1AwIhj2Cdj0z","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.776664_87.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.776664142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2kLGuYGaCkV1AwIhj2Cdj0z","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.875434_88.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.13","timestamp":"2021-02-08T07:18:40.875434170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_szU7g2F8bRR_qjRFAsjD8UV","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.880535_89.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.13","timestamp":"2021-02-08T07:18:40.880535006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_szU7g2F8bRR_qjRFAsjD8UV","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.882328_90.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.882328424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N9j0fBH4RG41rW2px2Kfg52","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.886694_91.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.886694720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N9j0fBH4RG41rW2px2Kfg52","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.942414_92.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.14","timestamp":"2021-02-08T07:18:40.942414834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c2MoBLNAFcZhCc0-6FkAure","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.947900_93.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.14","timestamp":"2021-02-08T07:18:40.947900056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c2MoBLNAFcZhCc0-6FkAure","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.950597_94.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.950597043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o5ykWPY7kYS85AigxFS37fZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.955189_95.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.955189582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o5ykWPY7kYS85AigxFS37fZ","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768721.049288_96.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.15","timestamp":"2021-02-08T07:18:41.049288715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KNldoUM05NwRop7PdH1FruD","ver":1},"status":{"status":"write"}}},{"id":"1_1612768721.055941_97.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.15","timestamp":"2021-02-08T07:18:41.055941239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KNldoUM05NwRop7PdH1FruD","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768721.058596_98.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:41.058596909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Zk1CJRJYFjyBIHzf8tu1OT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768721.063275_99.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:41.063275680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Zk1CJRJYFjyBIHzf8tu1OT","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768729.345437_100.1","section":"bucket.instance","name":"lhddgo-7:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16","timestamp":"2021-02-08T07:18:49.345437338Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_00nF2x8tkdYhe8Yvf2J_kXR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.351031_101.1","section":"bucket.instance","name":"lhddgo-7:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16","timestamp":"2021-02-08T07:18:49.351031227Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_00nF2x8tkdYhe8Yvf2J_kXR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.353169_102.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:49.353169069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.358754_103.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:49.358754345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.382909_104.1","section":"bucket.instance","name":"lhddgo-8:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17","timestamp":"2021-02-08T07:18:49.382909424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A2V06k6mnYpoHdwsmFO6I2c","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.389686_105.1","section":"bucket.instance","name":"lhddgo-8:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17","timestamp":"2021-02-08T07:18:49.389686951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A2V06k6mnYpoHdwsmFO6I2c","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.392255_106.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:49.392255869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.398757_107.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:49.398757052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.447313_108.1","section":"bucket.instance","name":"lhddgo-9:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18","timestamp":"2021-02-08T07:18:49.447313935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dG0OIO3lIHMoZH43A6aVRSG","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.453059_109.1","section":"bucket.instance","name":"lhddgo-9:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18","timestamp":"2021-02-08T07:18:49.453059961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dG0OIO3lIHMoZH43A6aVRSG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.455135_110.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:49.455135230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.462500_111.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:49.462500549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768738.338507_112.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:58.338507156Z","data":{"read_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768738.346128_113.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:58.346128189Z","data":{"read_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768738.734584_114.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:58.734584501Z","data":{"read_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768738.742706_115.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:58.742706390Z","data":{"read_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768739.744978_116.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:59.744978355Z","data":{"read_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768739.752484_117.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:59.752484319Z","data":{"read_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768746.954486_46.1","section":"bucket.instance","name":"lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19","timestamp":"2021-02-08T07:19:06.954486618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZ2_foaAGbx_i2Sm3CpW_Kk","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.960028_55.1","section":"bucket.instance","name":"lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19","timestamp":"2021-02-08T07:19:06.960028764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZ2_foaAGbx_i2Sm3CpW_Kk","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.961876_56.1","section":"bucket","name":"lhddgo-10","timestamp":"2021-02-08T07:19:06.961876770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PahWfqgiX7tHFTo7PfKIGym","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.966814_57.1","section":"bucket","name":"lhddgo-10","timestamp":"2021-02-08T07:19:06.966814969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PahWfqgiX7tHFTo7PfKIGym","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.988241_58.1","section":"bucket.instance","name":"lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20","timestamp":"2021-02-08T07:19:06.988241783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zC5o4T_ukEDV-6O241V0z4P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.993594_59.1","section":"bucket.instance","name":"lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20","timestamp":"2021-02-08T07:19:06.993594301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zC5o4T_ukEDV-6O241V0z4P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.995614_60.1","section":"bucket","name":"lhddgo-11","timestamp":"2021-02-08T07:19:06.995614191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lkJAfr_qEGLVNdY3mgxQHeC","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.000743_61.1","section":"bucket","name":"lhddgo-11","timestamp":"2021-02-08T07:19:07.000743900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lkJAfr_qEGLVNdY3mgxQHeC","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768747.045230_63.1","section":"bucket.instance","name":"lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21","timestamp":"2021-02-08T07:19:07.045230732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7eGXe-HdPcO7HYGGIubcFdo","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.050420_64.1","section":"bucket.instance","name":"lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21","timestamp":"2021-02-08T07:19:07.050420217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7eGXe-HdPcO7HYGGIubcFdo","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768747.052392_65.1","section":"bucket","name":"lhddgo-12","timestamp":"2021-02-08T07:19:07.052392440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_vX8F929OWDaiEf5eFNP9P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.057366_66.1","section":"bucket","name":"lhddgo-12","timestamp":"2021-02-08T07:19:07.057366611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_vX8F929OWDaiEf5eFNP9P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.456382_133.1","section":"bucket.instance","name":"lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22","timestamp":"2021-02-08T07:19:23.456382197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ttLcbiaMaOBic5ZoV20Gusq","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.462897_135.1","section":"bucket.instance","name":"lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22","timestamp":"2021-02-08T07:19:23.462897145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ttLcbiaMaOBic5ZoV20Gusq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.465037_136.1","section":"bucket","name":"lhddgo-13","timestamp":"2021-02-08T07:19:23.465037077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s0fnDUyDPo0ZxNuB6AgyP4P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.471695_137.1","section":"bucket","name":"lhddgo-13","timestamp":"2021-02-08T07:19:23.471695624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s0fnDUyDPo0ZxNuB6AgyP4P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.497377_138.1","section":"bucket.instance","name":"lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23","timestamp":"2021-02-08T07:19:23.497377314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f25x5uML91RHGgZY3mqvkiD","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.503198_139.1","section":"bucket.instance","name":"lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23","timestamp":"2021-02-08T07:19:23.503198102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f25x5uML91RHGgZY3mqvkiD","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.505240_140.1","section":"bucket","name":"lhddgo-14","timestamp":"2021-02-08T07:19:23.505240280Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yfbtCQsCHEWsWQtRDJWU0TK","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.510426_141.1","section":"bucket","name":"lhddgo-14","timestamp":"2021-02-08T07:19:23.510426290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yfbtCQsCHEWsWQtRDJWU0TK","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.556298_142.1","section":"bucket.instance","name":"lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24","timestamp":"2021-02-08T07:19:23.556298278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_86ctNhKnzUyhf-uA2VFMTRR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.563119_143.1","section":"bucket.instance","name":"lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24","timestamp":"2021-02-08T07:19:23.563119127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_86ctNhKnzUyhf-uA2VFMTRR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.565115_144.1","section":"bucket","name":"lhddgo-15","timestamp":"2021-02-08T07:19:23.565115398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kxIgLsiJU_eIuv-WaTRYw-R","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.570467_145.1","section":"bucket","name":"lhddgo-15","timestamp":"2021-02-08T07:19:23.570467263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kxIgLsiJU_eIuv-WaTRYw-R","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.007216_146.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.007216510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.012745_159.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.012745436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.014570_160.1","section":"bucket","name":"lhddgo-16","timestamp":"2021-02-08T07:20:12.014570514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nekdSdSgiiFVOM3sLjoHrs0","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.020148_161.1","section":"bucket","name":"lhddgo-16","timestamp":"2021-02-08T07:20:12.020148921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nekdSdSgiiFVOM3sLjoHrs0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.045071_162.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.045071037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.050744_163.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.050744782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.052634_164.1","section":"bucket","name":"lhddgo-17","timestamp":"2021-02-08T07:20:12.052634245Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__7j--ODbV4i8iflCZWp9f8W","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.057766_165.1","section":"bucket","name":"lhddgo-17","timestamp":"2021-02-08T07:20:12.057766474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__7j--ODbV4i8iflCZWp9f8W","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.102645_166.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.102645691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.108722_167.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.108722889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.110763_168.1","section":"bucket","name":"lhddgo-18","timestamp":"2021-02-08T07:20:12.110763801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zK0nzDVJpyF9AMlQzKt7CIi","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.116166_169.1","section":"bucket","name":"lhddgo-18","timestamp":"2021-02-08T07:20:12.116166136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zK0nzDVJpyF9AMlQzKt7CIi","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.139872_170.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.139872515Z","data":{"read_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.145372_171.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.145372779Z","data":{"read_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768812.149764_172.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.149764590Z","data":{"read_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.155292_173.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.155292011Z","data":{"read_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768812.165468_174.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.165468554Z","data":{"read_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.172314_175.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.172314820Z","data":{"read_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768909.754047_176.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.754047254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"status":{"status":"write"}}},{"id":"1_1612768909.759899_215.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.759899359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768909.761790_216.1","section":"bucket","name":"lhddgo-19","timestamp":"2021-02-08T07:21:49.761790823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7UU_grw1aDKDg8-iLh2ozDM","ver":1},"status":{"status":"write"}}},{"id":"1_1612768909.766466_217.1","section":"bucket","name":"lhddgo-19","timestamp":"2021-02-08T07:21:49.766466920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7UU_grw1aDKDg8-iLh2ozDM","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768909.772573_218.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.772573230Z","data":{"read_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":2},"status":{"status":"write"}}},{"id":"1_1612768909.778782_219.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.778782168Z","data":{"read_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768974.725779_233.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:22:54.725779997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"status":{"status":"write"}}},{"id":"1_1612768974.732118_239.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:22:54.732118058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768974.734076_240.1","section":"bucket","name":"lhddgo-20","timestamp":"2021-02-08T07:22:54.734076231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ewPFRqRbQtYiKfY1emF_0kh","ver":1},"status":{"status":"write"}}},{"id":"1_1612768974.739752_241.1","section":"bucket","name":"lhddgo-20","timestamp":"2021-02-08T07:22:54.739752084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ewPFRqRbQtYiKfY1emF_0kh","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769001.637403_242.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:21.637403627Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"status":{"status":"write"}}},{"id":"1_1612769001.644747_249.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:21.644747197Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769021.196864_250.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:41.196864849Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":3},"status":{"status":"write"}}},{"id":"1_1612769021.207297_257.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:41.207297516Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769042.799574_258.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.799574420Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.805549_265.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.805549492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.807942_266.1","section":"bucket","name":"lhddgo-21","timestamp":"2021-02-08T07:24:02.807942128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-4b49qybZ0UkKdoSGfItB4t","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.813228_267.1","section":"bucket","name":"lhddgo-21","timestamp":"2021-02-08T07:24:02.813228410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-4b49qybZ0UkKdoSGfItB4t","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.838690_268.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.838690572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.844449_269.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.844449108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.846408_270.1","section":"bucket","name":"lhddgo-22","timestamp":"2021-02-08T07:24:02.846408624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jUIV-7to8G5ZTs1obxvJjJG","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.851416_271.1","section":"bucket","name":"lhddgo-22","timestamp":"2021-02-08T07:24:02.851416259Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jUIV-7to8G5ZTs1obxvJjJG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.895189_272.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.895189936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.900758_273.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.900758211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.902745_274.1","section":"bucket","name":"lhddgo-23","timestamp":"2021-02-08T07:24:02.902745272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OjI4iZOJTWFwklM66d8DZiZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.907816_275.1","section":"bucket","name":"lhddgo-23","timestamp":"2021-02-08T07:24:02.907816855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OjI4iZOJTWFwklM66d8DZiZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.934689_276.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.934689328Z","data":{"read_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.942454_277.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.942454508Z","data":{"read_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769042.948058_278.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.948058553Z","data":{"read_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.956522_279.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.956522086Z","data":{"read_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769042.968947_280.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.968947604Z","data":{"read_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.978250_281.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.978250456Z","data":{"read_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.623875_282.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.623875923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.630835_299.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.630835843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.633069_300.1","section":"bucket","name":"lhddgo-24","timestamp":"2021-02-08T07:24:29.633069001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4t2fafA9W7emjovORdu4vzB","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.637883_301.1","section":"bucket","name":"lhddgo-24","timestamp":"2021-02-08T07:24:29.637883250Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4t2fafA9W7emjovORdu4vzB","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.660183_302.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.660183105Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.665900_303.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.665900158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.667859_304.1","section":"bucket","name":"lhddgo-25","timestamp":"2021-02-08T07:24:29.667859791Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5cBMeEkIR40rErwMHaeqFJ2","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.673799_305.1","section":"bucket","name":"lhddgo-25","timestamp":"2021-02-08T07:24:29.673799031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5cBMeEkIR40rErwMHaeqFJ2","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.720508_306.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.720508838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.726120_307.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.726120869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.727996_308.1","section":"bucket","name":"lhddgo-26","timestamp":"2021-02-08T07:24:29.727996128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__bV0SY4m7IJ85i318hYa99f","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.732935_309.1","section":"bucket","name":"lhddgo-26","timestamp":"2021-02-08T07:24:29.732935776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__bV0SY4m7IJ85i318hYa99f","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.767812_310.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.767812812Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.774076_311.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.774076941Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.778099_312.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.778099238Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.785267_313.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.785267130Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769069.790204_314.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.790204719Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.797562_315.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.797562958Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.809280_316.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.809280114Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.816992_317.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.816992209Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769069.828918_318.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.828918200Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.837623_319.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.837623812Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.849480_320.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.849480608Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.857413_321.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.857413152Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769098.920309_322.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:58.920309067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.927047_339.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:58.927047541Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.929208_340.1","section":"bucket","name":"lhddgo-27","timestamp":"2021-02-08T07:24:58.929208268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E3g7A0Syvz7sJwkowawNoEA","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.934262_341.1","section":"bucket","name":"lhddgo-27","timestamp":"2021-02-08T07:24:58.934262099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E3g7A0Syvz7sJwkowawNoEA","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.961964_342.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:58.961964655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.968667_343.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:58.968667208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.970843_344.1","section":"bucket","name":"lhddgo-28","timestamp":"2021-02-08T07:24:58.970843625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yXVDMAYob-zCi8EUwg1b8rK","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.976100_345.1","section":"bucket","name":"lhddgo-28","timestamp":"2021-02-08T07:24:58.976100609Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yXVDMAYob-zCi8EUwg1b8rK","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.021389_346.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.021389397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.027116_347.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.027116452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.029204_348.1","section":"bucket","name":"lhddgo-29","timestamp":"2021-02-08T07:24:59.029204248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BBdXyLdhOQBxsIZPgiWK1Io","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.040870_349.1","section":"bucket","name":"lhddgo-29","timestamp":"2021-02-08T07:24:59.040870109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BBdXyLdhOQBxsIZPgiWK1Io","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.070082_350.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:59.070082042Z","data":{"read_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.076291_351.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:59.076291919Z","data":{"read_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.128145_352.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:59.128145356Z","data":{"read_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.138654_353.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:59.138654984Z","data":{"read_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.195524_354.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.195524900Z","data":{"read_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.205763_355.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.205763967Z","data":{"read_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.283799_356.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.283799364Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.289172_357.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.289172513Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.291276_358.1","section":"bucket","name":"lhddgo-30","timestamp":"2021-02-08T07:24:59.291276569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UwjaMGoy_DrUnxaidzTazZd","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.296560_359.1","section":"bucket","name":"lhddgo-30","timestamp":"2021-02-08T07:24:59.296560718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UwjaMGoy_DrUnxaidzTazZd","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.322298_360.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.322298131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.328870_361.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.328870951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.331060_362.1","section":"bucket","name":"lhddgo-31","timestamp":"2021-02-08T07:24:59.331060564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2QZd7jhCZ6WbQQApzYZ6Zuq","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.336978_363.1","section":"bucket","name":"lhddgo-31","timestamp":"2021-02-08T07:24:59.336978349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2QZd7jhCZ6WbQQApzYZ6Zuq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.384623_364.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.384623961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.390826_365.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.390826118Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.392754_366.1","section":"bucket","name":"lhddgo-32","timestamp":"2021-02-08T07:24:59.392754749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bqm4fq2xDjXcNewXd8Ho5au","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.397952_367.1","section":"bucket","name":"lhddgo-32","timestamp":"2021-02-08T07:24:59.397952631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bqm4fq2xDjXcNewXd8Ho5au","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.425432_368.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.425432089Z","data":{"read_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.431859_369.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.431859354Z","data":{"read_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.481778_370.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.481778111Z","data":{"read_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.492185_371.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.492185468Z","data":{"read_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.548697_372.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.548697908Z","data":{"read_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.554553_373.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.554553520Z","data":{"read_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.625575_374.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.625575517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.632352_375.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.632352418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.634874_376.1","section":"bucket","name":"lhddgo-33","timestamp":"2021-02-08T07:24:59.634874165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S1nyQKgyKti5lpNsSFRpOE5","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.640734_377.1","section":"bucket","name":"lhddgo-33","timestamp":"2021-02-08T07:24:59.640734750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S1nyQKgyKti5lpNsSFRpOE5","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.671312_378.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:24:59.671312553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.679437_379.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:24:59.679437197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.681736_380.1","section":"bucket","name":"lhddgo-34","timestamp":"2021-02-08T07:24:59.681736519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BeJshNgmDKzGicFSITjzLQF","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.686854_381.1","section":"bucket","name":"lhddgo-34","timestamp":"2021-02-08T07:24:59.686854680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BeJshNgmDKzGicFSITjzLQF","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.734811_382.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:24:59.734811285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.740722_383.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:24:59.740722499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.742608_384.1","section":"bucket","name":"lhddgo-35","timestamp":"2021-02-08T07:24:59.742608646Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q-9lHc8CrdYvDkHWoFJqG8X","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.748237_385.1","section":"bucket","name":"lhddgo-35","timestamp":"2021-02-08T07:24:59.748237598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q-9lHc8CrdYvDkHWoFJqG8X","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.776781_386.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.776781310Z","data":{"read_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.783360_387.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.783360110Z","data":{"read_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769100.836953_388.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:25:00.836953168Z","data":{"read_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":2},"status":{"status":"write"}}},{"id":"1_1612769100.848578_389.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:25:00.848578115Z","data":{"read_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769102.144127_390.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:25:02.144127048Z","data":{"read_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":2},"status":{"status":"write"}}},{"id":"1_1612769102.156036_391.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:25:02.156036210Z","data":{"read_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769102.760731_392.1","section":"bucket.instance","name":"lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45","timestamp":"2021-02-08T07:25:02.760731916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pUmMyR0Isdm2KnONKSzA-Ey","ver":1},"status":{"status":"write"}}},{"id":"1_1612769102.767819_393.1","section":"bucket.instance","name":"lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45","timestamp":"2021-02-08T07:25:02.767819540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pUmMyR0Isdm2KnONKSzA-Ey","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769102.770229_394.1","section":"bucket","name":"lhddgo-36","timestamp":"2021-02-08T07:25:02.770229312Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8bkeVL24JNAUI6gG0schLP","ver":1},"status":{"status":"write"}}},{"id":"1_1612769102.775801_395.1","section":"bucket","name":"lhddgo-36","timestamp":"2021-02-08T07:25:02.775801188Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8bkeVL24JNAUI6gG0schLP","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.117151_396.1","section":"bucket.instance","name":"lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46","timestamp":"2021-02-08T07:25:03.117151764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g5Dq8aX87ZwjcUVPxfbcfaL","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.122644_397.1","section":"bucket.instance","name":"lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46","timestamp":"2021-02-08T07:25:03.122644962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g5Dq8aX87ZwjcUVPxfbcfaL","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.124978_398.1","section":"bucket","name":"lhddgo-37","timestamp":"2021-02-08T07:25:03.124978224Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__2oxvZ6RrkXDCfhasSdxBZQ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.130437_399.1","section":"bucket","name":"lhddgo-37","timestamp":"2021-02-08T07:25:03.130437751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__2oxvZ6RrkXDCfhasSdxBZQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.182592_400.1","section":"bucket.instance","name":"lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47","timestamp":"2021-02-08T07:25:03.182592377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_06zd8oUyALlG1ZsQD8Fdol-","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.187922_401.1","section":"bucket.instance","name":"lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47","timestamp":"2021-02-08T07:25:03.187922879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_06zd8oUyALlG1ZsQD8Fdol-","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.189848_402.1","section":"bucket","name":"lhddgo-38","timestamp":"2021-02-08T07:25:03.189848323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUOpEUVHAB91BG5u5pJNnrO","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.194698_403.1","section":"bucket","name":"lhddgo-38","timestamp":"2021-02-08T07:25:03.194698092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUOpEUVHAB91BG5u5pJNnrO","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769106.694646_404.1","section":"bucket.instance","name":"lhddgo-39:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48","timestamp":"2021-02-08T07:25:06.694646856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CNAF5lCyovQyVl-qG7T3ANF","ver":1},"status":{"status":"write"}}},{"id":"1_1612769106.699588_406.1","section":"bucket.instance","name":"lhddgo-39:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48","timestamp":"2021-02-08T07:25:06.699588368Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CNAF5lCyovQyVl-qG7T3ANF","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769106.701410_407.1","section":"bucket","name":"lhddgo-39","timestamp":"2021-02-08T07:25:06.701410302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1SwIrqXTWHi7vifEqt55SsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769106.705845_408.1","section":"bucket","name":"lhddgo-39","timestamp":"2021-02-08T07:25:06.705845190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1SwIrqXTWHi7vifEqt55SsQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.036501_409.1","section":"bucket.instance","name":"lhddgo-40:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49","timestamp":"2021-02-08T07:25:07.036501974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dDggL3R9PYxxraDauffrO09","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.042000_410.1","section":"bucket.instance","name":"lhddgo-40:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49","timestamp":"2021-02-08T07:25:07.042000544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dDggL3R9PYxxraDauffrO09","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.043987_411.1","section":"bucket","name":"lhddgo-40","timestamp":"2021-02-08T07:25:07.043987355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZOZ_VlHozV1b8Q-DrynRt61","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.049028_412.1","section":"bucket","name":"lhddgo-40","timestamp":"2021-02-08T07:25:07.049028688Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZOZ_VlHozV1b8Q-DrynRt61","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.097668_413.1","section":"bucket.instance","name":"lhddgo-41:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50","timestamp":"2021-02-08T07:25:07.097668057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTrt6zeu476exuMMzu3hcGZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.103981_414.1","section":"bucket.instance","name":"lhddgo-41:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50","timestamp":"2021-02-08T07:25:07.103981325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTrt6zeu476exuMMzu3hcGZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.106208_415.1","section":"bucket","name":"lhddgo-41","timestamp":"2021-02-08T07:25:07.106208500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YtG89TnLF4vYQMJMWzkgy3J","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.111803_416.1","section":"bucket","name":"lhddgo-41","timestamp":"2021-02-08T07:25:07.111803603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YtG89TnLF4vYQMJMWzkgy3J","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:40.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:40.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:40.984 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period d4b11cd5-1a7c-4383-afcf-402844a1791d --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:41.022 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:41.023 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:41.280 INFO:teuthology.orchestra.run.smithi110.stdout:[{"id":"1_1612768579.522956_30.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-08T07:16:19.522956903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GTmvG-EF-9cF89V4myGNcR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768579.532632_32.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-08T07:16:19.532632715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GTmvG-EF-9cF89V4myGNcR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.649968_40.1","section":"bucket.instance","name":"lhddgo-1:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1","timestamp":"2021-02-08T07:18:21.649968470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e2Im8Aginrg7vTQ5Uq0UExs","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.675446_41.1","section":"bucket.instance","name":"lhddgo-1:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1","timestamp":"2021-02-08T07:18:21.675446655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e2Im8Aginrg7vTQ5Uq0UExs","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.677384_42.1","section":"bucket","name":"lhddgo-1","timestamp":"2021-02-08T07:18:21.677384640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m_Fn4BKOiM5g7mlQklEvF8h","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.681366_43.1","section":"bucket","name":"lhddgo-1","timestamp":"2021-02-08T07:18:21.681366592Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m_Fn4BKOiM5g7mlQklEvF8h","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.702503_44.1","section":"bucket.instance","name":"lhddgo-2:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2","timestamp":"2021-02-08T07:18:21.702503042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6qX65q9Zll4vnVd5kG0CkcV","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.708160_45.1","section":"bucket.instance","name":"lhddgo-2:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2","timestamp":"2021-02-08T07:18:21.708160313Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6qX65q9Zll4vnVd5kG0CkcV","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.710147_46.1","section":"bucket","name":"lhddgo-2","timestamp":"2021-02-08T07:18:21.710147097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1DMS1mNFwKdoHY2cnmaj9tt","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.714859_47.1","section":"bucket","name":"lhddgo-2","timestamp":"2021-02-08T07:18:21.714859854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1DMS1mNFwKdoHY2cnmaj9tt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768704.880852_48.1","section":"bucket.instance","name":"lhddgo-3:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3","timestamp":"2021-02-08T07:18:24.880852494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rOLGVtTLt3ajM6sppBkkd3e","ver":1},"status":{"status":"write"}}},{"id":"1_1612768704.886356_49.1","section":"bucket.instance","name":"lhddgo-3:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3","timestamp":"2021-02-08T07:18:24.886356317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rOLGVtTLt3ajM6sppBkkd3e","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768704.888224_50.1","section":"bucket","name":"lhddgo-3","timestamp":"2021-02-08T07:18:24.888224742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqx6QY3WLExCkHpK84fAXxr","ver":1},"status":{"status":"write"}}},{"id":"1_1612768704.893108_51.1","section":"bucket","name":"lhddgo-3","timestamp":"2021-02-08T07:18:24.893108028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqx6QY3WLExCkHpK84fAXxr","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.777993_52.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4","timestamp":"2021-02-08T07:18:32.777993072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BoYtQApnX7_2AtGejxpYaZO","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.783478_53.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4","timestamp":"2021-02-08T07:18:32.783478935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BoYtQApnX7_2AtGejxpYaZO","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.785319_54.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:32.785319651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fp7ohDPD_24JfOiBzo-rYXf","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.791923_55.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:32.791923400Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fp7ohDPD_24JfOiBzo-rYXf","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.814845_56.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5","timestamp":"2021-02-08T07:18:32.814845373Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4CPxPuDJA5f6reOzSejIeq","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.820494_57.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5","timestamp":"2021-02-08T07:18:32.820494612Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4CPxPuDJA5f6reOzSejIeq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.822303_58.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:32.822303238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_yHBtAEVD0Gyl9lvsZLfTL","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.830478_59.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:32.830478754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_yHBtAEVD0Gyl9lvsZLfTL","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.875274_60.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6","timestamp":"2021-02-08T07:18:32.875274066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIAw4BnGrFg8-fkZKIn5AkT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.883030_61.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6","timestamp":"2021-02-08T07:18:32.883030645Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIAw4BnGrFg8-fkZKIn5AkT","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.885168_62.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:32.885168872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hS--GkAzPDMbDD0Ah6ChxD9","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.891808_63.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:32.891808199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hS--GkAzPDMbDD0Ah6ChxD9","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.455112_64.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.7","timestamp":"2021-02-08T07:18:40.455112453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D64Kh1AQ6YPfhYh6ce23ln9","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.461143_65.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.7","timestamp":"2021-02-08T07:18:40.461143164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D64Kh1AQ6YPfhYh6ce23ln9","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.463040_66.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.463040385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1HoTcP5PU2Kgj55NSIKnwiu","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.467993_67.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.467993630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1HoTcP5PU2Kgj55NSIKnwiu","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.499332_68.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.8","timestamp":"2021-02-08T07:18:40.499332959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ghr6TovGAVCrhaoBiwCDcI0","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.504921_69.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.8","timestamp":"2021-02-08T07:18:40.504921848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ghr6TovGAVCrhaoBiwCDcI0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.507224_70.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.507224631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRWiK6cMflxrN6xErwMtL8C","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.511355_71.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.511355956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRWiK6cMflxrN6xErwMtL8C","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.543636_72.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.9","timestamp":"2021-02-08T07:18:40.543636213Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xlBzXd-b99YpCwUbXyq6T_I","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.549664_73.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.9","timestamp":"2021-02-08T07:18:40.549664095Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xlBzXd-b99YpCwUbXyq6T_I","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.551977_74.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.551977841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2tIbTzLnZNEgqsoMi4Kyu4g","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.556906_75.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.556906173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2tIbTzLnZNEgqsoMi4Kyu4g","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.598520_76.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.10","timestamp":"2021-02-08T07:18:40.598520764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_erUTYNTZcG6PubXCSk30xX6","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.603500_77.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.10","timestamp":"2021-02-08T07:18:40.603500331Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_erUTYNTZcG6PubXCSk30xX6","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.605348_78.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.605348512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3KtUz4rocFNL2_vS5Rfub48","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.608562_79.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.608562308Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3KtUz4rocFNL2_vS5Rfub48","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.654740_80.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.11","timestamp":"2021-02-08T07:18:40.654740480Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HT1xmRKztbydlE5i670276O","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.660209_81.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.11","timestamp":"2021-02-08T07:18:40.660209900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HT1xmRKztbydlE5i670276O","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.662252_82.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.662252850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7sCGVKD50Piys06LRi5mRnA","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.667287_83.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.667287114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7sCGVKD50Piys06LRi5mRnA","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.764760_84.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.12","timestamp":"2021-02-08T07:18:40.764760372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vhvVK_HjSPWSrSoiQ3B1sWk","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.770576_85.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.12","timestamp":"2021-02-08T07:18:40.770576911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vhvVK_HjSPWSrSoiQ3B1sWk","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.772523_86.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.772523598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2kLGuYGaCkV1AwIhj2Cdj0z","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.776664_87.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.776664142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2kLGuYGaCkV1AwIhj2Cdj0z","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.875434_88.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.13","timestamp":"2021-02-08T07:18:40.875434170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_szU7g2F8bRR_qjRFAsjD8UV","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.880535_89.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.13","timestamp":"2021-02-08T07:18:40.880535006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_szU7g2F8bRR_qjRFAsjD8UV","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.882328_90.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.882328424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N9j0fBH4RG41rW2px2Kfg52","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.886694_91.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.886694720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N9j0fBH4RG41rW2px2Kfg52","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.942414_92.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.14","timestamp":"2021-02-08T07:18:40.942414834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c2MoBLNAFcZhCc0-6FkAure","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.947900_93.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.14","timestamp":"2021-02-08T07:18:40.947900056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c2MoBLNAFcZhCc0-6FkAure","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.950597_94.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.950597043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o5ykWPY7kYS85AigxFS37fZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.955189_95.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.955189582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o5ykWPY7kYS85AigxFS37fZ","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768721.049288_96.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.15","timestamp":"2021-02-08T07:18:41.049288715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KNldoUM05NwRop7PdH1FruD","ver":1},"status":{"status":"write"}}},{"id":"1_1612768721.055941_97.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.15","timestamp":"2021-02-08T07:18:41.055941239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KNldoUM05NwRop7PdH1FruD","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768721.058596_98.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:41.058596909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Zk1CJRJYFjyBIHzf8tu1OT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768721.063275_99.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:41.063275680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Zk1CJRJYFjyBIHzf8tu1OT","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768729.345437_100.1","section":"bucket.instance","name":"lhddgo-7:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16","timestamp":"2021-02-08T07:18:49.345437338Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_00nF2x8tkdYhe8Yvf2J_kXR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.351031_101.1","section":"bucket.instance","name":"lhddgo-7:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16","timestamp":"2021-02-08T07:18:49.351031227Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_00nF2x8tkdYhe8Yvf2J_kXR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.353169_102.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:49.353169069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.358754_103.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:49.358754345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.382909_104.1","section":"bucket.instance","name":"lhddgo-8:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17","timestamp":"2021-02-08T07:18:49.382909424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A2V06k6mnYpoHdwsmFO6I2c","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.389686_105.1","section":"bucket.instance","name":"lhddgo-8:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17","timestamp":"2021-02-08T07:18:49.389686951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A2V06k6mnYpoHdwsmFO6I2c","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.392255_106.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:49.392255869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.398757_107.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:49.398757052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.447313_108.1","section":"bucket.instance","name":"lhddgo-9:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18","timestamp":"2021-02-08T07:18:49.447313935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dG0OIO3lIHMoZH43A6aVRSG","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.453059_109.1","section":"bucket.instance","name":"lhddgo-9:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18","timestamp":"2021-02-08T07:18:49.453059961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dG0OIO3lIHMoZH43A6aVRSG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.455135_110.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:49.455135230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.462500_111.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:49.462500549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768738.338507_112.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:58.338507156Z","data":{"read_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768738.346128_113.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:58.346128189Z","data":{"read_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768738.734584_114.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:58.734584501Z","data":{"read_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768738.742706_115.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:58.742706390Z","data":{"read_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768739.744978_116.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:59.744978355Z","data":{"read_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768739.752484_117.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:59.752484319Z","data":{"read_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768746.954486_46.1","section":"bucket.instance","name":"lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19","timestamp":"2021-02-08T07:19:06.954486618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZ2_foaAGbx_i2Sm3CpW_Kk","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.960028_55.1","section":"bucket.instance","name":"lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19","timestamp":"2021-02-08T07:19:06.960028764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZ2_foaAGbx_i2Sm3CpW_Kk","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.961876_56.1","section":"bucket","name":"lhddgo-10","timestamp":"2021-02-08T07:19:06.961876770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PahWfqgiX7tHFTo7PfKIGym","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.966814_57.1","section":"bucket","name":"lhddgo-10","timestamp":"2021-02-08T07:19:06.966814969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PahWfqgiX7tHFTo7PfKIGym","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.988241_58.1","section":"bucket.instance","name":"lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20","timestamp":"2021-02-08T07:19:06.988241783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zC5o4T_ukEDV-6O241V0z4P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.993594_59.1","section":"bucket.instance","name":"lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20","timestamp":"2021-02-08T07:19:06.993594301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zC5o4T_ukEDV-6O241V0z4P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.995614_60.1","section":"bucket","name":"lhddgo-11","timestamp":"2021-02-08T07:19:06.995614191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lkJAfr_qEGLVNdY3mgxQHeC","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.000743_61.1","section":"bucket","name":"lhddgo-11","timestamp":"2021-02-08T07:19:07.000743900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lkJAfr_qEGLVNdY3mgxQHeC","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768747.045230_63.1","section":"bucket.instance","name":"lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21","timestamp":"2021-02-08T07:19:07.045230732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7eGXe-HdPcO7HYGGIubcFdo","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.050420_64.1","section":"bucket.instance","name":"lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21","timestamp":"2021-02-08T07:19:07.050420217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7eGXe-HdPcO7HYGGIubcFdo","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768747.052392_65.1","section":"bucket","name":"lhddgo-12","timestamp":"2021-02-08T07:19:07.052392440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_vX8F929OWDaiEf5eFNP9P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.057366_66.1","section":"bucket","name":"lhddgo-12","timestamp":"2021-02-08T07:19:07.057366611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_vX8F929OWDaiEf5eFNP9P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.456382_133.1","section":"bucket.instance","name":"lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22","timestamp":"2021-02-08T07:19:23.456382197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ttLcbiaMaOBic5ZoV20Gusq","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.462897_135.1","section":"bucket.instance","name":"lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22","timestamp":"2021-02-08T07:19:23.462897145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ttLcbiaMaOBic5ZoV20Gusq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.465037_136.1","section":"bucket","name":"lhddgo-13","timestamp":"2021-02-08T07:19:23.465037077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s0fnDUyDPo0ZxNuB6AgyP4P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.471695_137.1","section":"bucket","name":"lhddgo-13","timestamp":"2021-02-08T07:19:23.471695624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s0fnDUyDPo0ZxNuB6AgyP4P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.497377_138.1","section":"bucket.instance","name":"lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23","timestamp":"2021-02-08T07:19:23.497377314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f25x5uML91RHGgZY3mqvkiD","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.503198_139.1","section":"bucket.instance","name":"lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23","timestamp":"2021-02-08T07:19:23.503198102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f25x5uML91RHGgZY3mqvkiD","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.505240_140.1","section":"bucket","name":"lhddgo-14","timestamp":"2021-02-08T07:19:23.505240280Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yfbtCQsCHEWsWQtRDJWU0TK","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.510426_141.1","section":"bucket","name":"lhddgo-14","timestamp":"2021-02-08T07:19:23.510426290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yfbtCQsCHEWsWQtRDJWU0TK","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.556298_142.1","section":"bucket.instance","name":"lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24","timestamp":"2021-02-08T07:19:23.556298278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_86ctNhKnzUyhf-uA2VFMTRR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.563119_143.1","section":"bucket.instance","name":"lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24","timestamp":"2021-02-08T07:19:23.563119127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_86ctNhKnzUyhf-uA2VFMTRR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.565115_144.1","section":"bucket","name":"lhddgo-15","timestamp":"2021-02-08T07:19:23.565115398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kxIgLsiJU_eIuv-WaTRYw-R","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.570467_145.1","section":"bucket","name":"lhddgo-15","timestamp":"2021-02-08T07:19:23.570467263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kxIgLsiJU_eIuv-WaTRYw-R","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.007216_146.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.007216510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.012745_159.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.012745436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.014570_160.1","section":"bucket","name":"lhddgo-16","timestamp":"2021-02-08T07:20:12.014570514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nekdSdSgiiFVOM3sLjoHrs0","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.020148_161.1","section":"bucket","name":"lhddgo-16","timestamp":"2021-02-08T07:20:12.020148921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nekdSdSgiiFVOM3sLjoHrs0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.045071_162.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.045071037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.050744_163.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.050744782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.052634_164.1","section":"bucket","name":"lhddgo-17","timestamp":"2021-02-08T07:20:12.052634245Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__7j--ODbV4i8iflCZWp9f8W","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.057766_165.1","section":"bucket","name":"lhddgo-17","timestamp":"2021-02-08T07:20:12.057766474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__7j--ODbV4i8iflCZWp9f8W","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.102645_166.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.102645691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.108722_167.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.108722889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.110763_168.1","section":"bucket","name":"lhddgo-18","timestamp":"2021-02-08T07:20:12.110763801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zK0nzDVJpyF9AMlQzKt7CIi","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.116166_169.1","section":"bucket","name":"lhddgo-18","timestamp":"2021-02-08T07:20:12.116166136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zK0nzDVJpyF9AMlQzKt7CIi","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.139872_170.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.139872515Z","data":{"read_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.145372_171.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.145372779Z","data":{"read_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768812.149764_172.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.149764590Z","data":{"read_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.155292_173.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.155292011Z","data":{"read_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768812.165468_174.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.165468554Z","data":{"read_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.172314_175.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.172314820Z","data":{"read_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768909.754047_176.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.754047254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"status":{"status":"write"}}},{"id":"1_1612768909.759899_215.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.759899359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768909.761790_216.1","section":"bucket","name":"lhddgo-19","timestamp":"2021-02-08T07:21:49.761790823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7UU_grw1aDKDg8-iLh2ozDM","ver":1},"status":{"status":"write"}}},{"id":"1_1612768909.766466_217.1","section":"bucket","name":"lhddgo-19","timestamp":"2021-02-08T07:21:49.766466920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7UU_grw1aDKDg8-iLh2ozDM","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768909.772573_218.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.772573230Z","data":{"read_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":2},"status":{"status":"write"}}},{"id":"1_1612768909.778782_219.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.778782168Z","data":{"read_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768974.725779_233.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:22:54.725779997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"status":{"status":"write"}}},{"id":"1_1612768974.732118_239.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:22:54.732118058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768974.734076_240.1","section":"bucket","name":"lhddgo-20","timestamp":"2021-02-08T07:22:54.734076231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ewPFRqRbQtYiKfY1emF_0kh","ver":1},"status":{"status":"write"}}},{"id":"1_1612768974.739752_241.1","section":"bucket","name":"lhddgo-20","timestamp":"2021-02-08T07:22:54.739752084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ewPFRqRbQtYiKfY1emF_0kh","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769001.637403_242.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:21.637403627Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"status":{"status":"write"}}},{"id":"1_1612769001.644747_249.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:21.644747197Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769021.196864_250.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:41.196864849Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":3},"status":{"status":"write"}}},{"id":"1_1612769021.207297_257.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:41.207297516Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769042.799574_258.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.799574420Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.805549_265.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.805549492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.807942_266.1","section":"bucket","name":"lhddgo-21","timestamp":"2021-02-08T07:24:02.807942128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-4b49qybZ0UkKdoSGfItB4t","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.813228_267.1","section":"bucket","name":"lhddgo-21","timestamp":"2021-02-08T07:24:02.813228410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-4b49qybZ0UkKdoSGfItB4t","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.838690_268.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.838690572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.844449_269.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.844449108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.846408_270.1","section":"bucket","name":"lhddgo-22","timestamp":"2021-02-08T07:24:02.846408624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jUIV-7to8G5ZTs1obxvJjJG","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.851416_271.1","section":"bucket","name":"lhddgo-22","timestamp":"2021-02-08T07:24:02.851416259Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jUIV-7to8G5ZTs1obxvJjJG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.895189_272.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.895189936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.900758_273.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.900758211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.902745_274.1","section":"bucket","name":"lhddgo-23","timestamp":"2021-02-08T07:24:02.902745272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OjI4iZOJTWFwklM66d8DZiZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.907816_275.1","section":"bucket","name":"lhddgo-23","timestamp":"2021-02-08T07:24:02.907816855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OjI4iZOJTWFwklM66d8DZiZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.934689_276.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.934689328Z","data":{"read_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.942454_277.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.942454508Z","data":{"read_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769042.948058_278.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.948058553Z","data":{"read_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.956522_279.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.956522086Z","data":{"read_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769042.968947_280.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.968947604Z","data":{"read_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.978250_281.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.978250456Z","data":{"read_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.623875_282.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.623875923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.630835_299.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.630835843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.633069_300.1","section":"bucket","name":"lhddgo-24","timestamp":"2021-02-08T07:24:29.633069001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4t2fafA9W7emjovORdu4vzB","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.637883_301.1","section":"bucket","name":"lhddgo-24","timestamp":"2021-02-08T07:24:29.637883250Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4t2fafA9W7emjovORdu4vzB","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.660183_302.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.660183105Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.665900_303.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.665900158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.667859_304.1","section":"bucket","name":"lhddgo-25","timestamp":"2021-02-08T07:24:29.667859791Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5cBMeEkIR40rErwMHaeqFJ2","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.673799_305.1","section":"bucket","name":"lhddgo-25","timestamp":"2021-02-08T07:24:29.673799031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5cBMeEkIR40rErwMHaeqFJ2","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.720508_306.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.720508838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.726120_307.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.726120869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.727996_308.1","section":"bucket","name":"lhddgo-26","timestamp":"2021-02-08T07:24:29.727996128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__bV0SY4m7IJ85i318hYa99f","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.732935_309.1","section":"bucket","name":"lhddgo-26","timestamp":"2021-02-08T07:24:29.732935776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__bV0SY4m7IJ85i318hYa99f","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.767812_310.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.767812812Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.774076_311.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.774076941Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.778099_312.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.778099238Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.785267_313.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.785267130Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769069.790204_314.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.790204719Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.797562_315.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.797562958Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.809280_316.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.809280114Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.816992_317.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.816992209Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769069.828918_318.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.828918200Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.837623_319.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.837623812Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.849480_320.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.849480608Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.857413_321.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.857413152Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769098.920309_322.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:58.920309067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.927047_339.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:58.927047541Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.929208_340.1","section":"bucket","name":"lhddgo-27","timestamp":"2021-02-08T07:24:58.929208268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E3g7A0Syvz7sJwkowawNoEA","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.934262_341.1","section":"bucket","name":"lhddgo-27","timestamp":"2021-02-08T07:24:58.934262099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E3g7A0Syvz7sJwkowawNoEA","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.961964_342.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:58.961964655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.968667_343.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:58.968667208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.970843_344.1","section":"bucket","name":"lhddgo-28","timestamp":"2021-02-08T07:24:58.970843625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yXVDMAYob-zCi8EUwg1b8rK","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.976100_345.1","section":"bucket","name":"lhddgo-28","timestamp":"2021-02-08T07:24:58.976100609Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yXVDMAYob-zCi8EUwg1b8rK","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.021389_346.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.021389397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.027116_347.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.027116452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.029204_348.1","section":"bucket","name":"lhddgo-29","timestamp":"2021-02-08T07:24:59.029204248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BBdXyLdhOQBxsIZPgiWK1Io","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.040870_349.1","section":"bucket","name":"lhddgo-29","timestamp":"2021-02-08T07:24:59.040870109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BBdXyLdhOQBxsIZPgiWK1Io","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.070082_350.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:59.070082042Z","data":{"read_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.076291_351.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:59.076291919Z","data":{"read_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.128145_352.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:59.128145356Z","data":{"read_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.138654_353.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:59.138654984Z","data":{"read_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.195524_354.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.195524900Z","data":{"read_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.205763_355.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.205763967Z","data":{"read_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.283799_356.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.283799364Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.289172_357.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.289172513Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.291276_358.1","section":"bucket","name":"lhddgo-30","timestamp":"2021-02-08T07:24:59.291276569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UwjaMGoy_DrUnxaidzTazZd","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.296560_359.1","section":"bucket","name":"lhddgo-30","timestamp":"2021-02-08T07:24:59.296560718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UwjaMGoy_DrUnxaidzTazZd","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.322298_360.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.322298131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.328870_361.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.328870951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.331060_362.1","section":"bucket","name":"lhddgo-31","timestamp":"2021-02-08T07:24:59.331060564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2QZd7jhCZ6WbQQApzYZ6Zuq","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.336978_363.1","section":"bucket","name":"lhddgo-31","timestamp":"2021-02-08T07:24:59.336978349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2QZd7jhCZ6WbQQApzYZ6Zuq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.384623_364.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.384623961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.390826_365.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.390826118Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.392754_366.1","section":"bucket","name":"lhddgo-32","timestamp":"2021-02-08T07:24:59.392754749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bqm4fq2xDjXcNewXd8Ho5au","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.397952_367.1","section":"bucket","name":"lhddgo-32","timestamp":"2021-02-08T07:24:59.397952631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bqm4fq2xDjXcNewXd8Ho5au","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.425432_368.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.425432089Z","data":{"read_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.431859_369.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.431859354Z","data":{"read_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.481778_370.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.481778111Z","data":{"read_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.492185_371.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.492185468Z","data":{"read_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.548697_372.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.548697908Z","data":{"read_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.554553_373.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.554553520Z","data":{"read_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.625575_374.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.625575517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.632352_375.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.632352418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.634874_376.1","section":"bucket","name":"lhddgo-33","timestamp":"2021-02-08T07:24:59.634874165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S1nyQKgyKti5lpNsSFRpOE5","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.640734_377.1","section":"bucket","name":"lhddgo-33","timestamp":"2021-02-08T07:24:59.640734750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S1nyQKgyKti5lpNsSFRpOE5","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.671312_378.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:24:59.671312553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.679437_379.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:24:59.679437197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.681736_380.1","section":"bucket","name":"lhddgo-34","timestamp":"2021-02-08T07:24:59.681736519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BeJshNgmDKzGicFSITjzLQF","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.686854_381.1","section":"bucket","name":"lhddgo-34","timestamp":"2021-02-08T07:24:59.686854680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BeJshNgmDKzGicFSITjzLQF","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.734811_382.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:24:59.734811285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.740722_383.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:24:59.740722499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.742608_384.1","section":"bucket","name":"lhddgo-35","timestamp":"2021-02-08T07:24:59.742608646Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q-9lHc8CrdYvDkHWoFJqG8X","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.748237_385.1","section":"bucket","name":"lhddgo-35","timestamp":"2021-02-08T07:24:59.748237598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q-9lHc8CrdYvDkHWoFJqG8X","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.776781_386.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.776781310Z","data":{"read_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.783360_387.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.783360110Z","data":{"read_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769100.836953_388.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:25:00.836953168Z","data":{"read_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":2},"status":{"status":"write"}}},{"id":"1_1612769100.848578_389.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:25:00.848578115Z","data":{"read_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769102.144127_390.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:25:02.144127048Z","data":{"read_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":2},"status":{"status":"write"}}},{"id":"1_1612769102.156036_391.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:25:02.156036210Z","data":{"read_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769102.760731_392.1","section":"bucket.instance","name":"lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45","timestamp":"2021-02-08T07:25:02.760731916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pUmMyR0Isdm2KnONKSzA-Ey","ver":1},"status":{"status":"write"}}},{"id":"1_1612769102.767819_393.1","section":"bucket.instance","name":"lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45","timestamp":"2021-02-08T07:25:02.767819540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pUmMyR0Isdm2KnONKSzA-Ey","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769102.770229_394.1","section":"bucket","name":"lhddgo-36","timestamp":"2021-02-08T07:25:02.770229312Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8bkeVL24JNAUI6gG0schLP","ver":1},"status":{"status":"write"}}},{"id":"1_1612769102.775801_395.1","section":"bucket","name":"lhddgo-36","timestamp":"2021-02-08T07:25:02.775801188Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8bkeVL24JNAUI6gG0schLP","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.117151_396.1","section":"bucket.instance","name":"lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46","timestamp":"2021-02-08T07:25:03.117151764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g5Dq8aX87ZwjcUVPxfbcfaL","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.122644_397.1","section":"bucket.instance","name":"lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46","timestamp":"2021-02-08T07:25:03.122644962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g5Dq8aX87ZwjcUVPxfbcfaL","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.124978_398.1","section":"bucket","name":"lhddgo-37","timestamp":"2021-02-08T07:25:03.124978224Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__2oxvZ6RrkXDCfhasSdxBZQ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.130437_399.1","section":"bucket","name":"lhddgo-37","timestamp":"2021-02-08T07:25:03.130437751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__2oxvZ6RrkXDCfhasSdxBZQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.182592_400.1","section":"bucket.instance","name":"lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47","timestamp":"2021-02-08T07:25:03.182592377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_06zd8oUyALlG1ZsQD8Fdol-","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.187922_401.1","section":"bucket.instance","name":"lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47","timestamp":"2021-02-08T07:25:03.187922879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_06zd8oUyALlG1ZsQD8Fdol-","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.189848_402.1","section":"bucket","name":"lhddgo-38","timestamp":"2021-02-08T07:25:03.189848323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUOpEUVHAB91BG5u5pJNnrO","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.194698_403.1","section":"bucket","name":"lhddgo-38","timestamp":"2021-02-08T07:25:03.194698092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUOpEUVHAB91BG5u5pJNnrO","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769106.694646_404.1","section":"bucket.instance","name":"lhddgo-39:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48","timestamp":"2021-02-08T07:25:06.694646856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CNAF5lCyovQyVl-qG7T3ANF","ver":1},"status":{"status":"write"}}},{"id":"1_1612769106.699588_406.1","section":"bucket.instance","name":"lhddgo-39:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48","timestamp":"2021-02-08T07:25:06.699588368Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CNAF5lCyovQyVl-qG7T3ANF","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769106.701410_407.1","section":"bucket","name":"lhddgo-39","timestamp":"2021-02-08T07:25:06.701410302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1SwIrqXTWHi7vifEqt55SsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769106.705845_408.1","section":"bucket","name":"lhddgo-39","timestamp":"2021-02-08T07:25:06.705845190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1SwIrqXTWHi7vifEqt55SsQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.036501_409.1","section":"bucket.instance","name":"lhddgo-40:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49","timestamp":"2021-02-08T07:25:07.036501974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dDggL3R9PYxxraDauffrO09","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.042000_410.1","section":"bucket.instance","name":"lhddgo-40:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49","timestamp":"2021-02-08T07:25:07.042000544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dDggL3R9PYxxraDauffrO09","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.043987_411.1","section":"bucket","name":"lhddgo-40","timestamp":"2021-02-08T07:25:07.043987355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZOZ_VlHozV1b8Q-DrynRt61","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.049028_412.1","section":"bucket","name":"lhddgo-40","timestamp":"2021-02-08T07:25:07.049028688Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZOZ_VlHozV1b8Q-DrynRt61","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.097668_413.1","section":"bucket.instance","name":"lhddgo-41:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50","timestamp":"2021-02-08T07:25:07.097668057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTrt6zeu476exuMMzu3hcGZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.103981_414.1","section":"bucket.instance","name":"lhddgo-41:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50","timestamp":"2021-02-08T07:25:07.103981325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTrt6zeu476exuMMzu3hcGZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.106208_415.1","section":"bucket","name":"lhddgo-41","timestamp":"2021-02-08T07:25:07.106208500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YtG89TnLF4vYQMJMWzkgy3J","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.111803_416.1","section":"bucket","name":"lhddgo-41","timestamp":"2021-02-08T07:25:07.111803603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YtG89TnLF4vYQMJMWzkgy3J","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:41.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:41.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:41.282 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period d4b11cd5-1a7c-4383-afcf-402844a1791d --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:41.322 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:41.322 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:41.635 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1612768579.522956_30.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-08T07:16:19.522956903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GTmvG-EF-9cF89V4myGNcR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768579.532632_32.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-08T07:16:19.532632715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GTmvG-EF-9cF89V4myGNcR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.649968_40.1","section":"bucket.instance","name":"lhddgo-1:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1","timestamp":"2021-02-08T07:18:21.649968470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e2Im8Aginrg7vTQ5Uq0UExs","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.675446_41.1","section":"bucket.instance","name":"lhddgo-1:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1","timestamp":"2021-02-08T07:18:21.675446655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e2Im8Aginrg7vTQ5Uq0UExs","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.677384_42.1","section":"bucket","name":"lhddgo-1","timestamp":"2021-02-08T07:18:21.677384640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m_Fn4BKOiM5g7mlQklEvF8h","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.681366_43.1","section":"bucket","name":"lhddgo-1","timestamp":"2021-02-08T07:18:21.681366592Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m_Fn4BKOiM5g7mlQklEvF8h","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.702503_44.1","section":"bucket.instance","name":"lhddgo-2:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2","timestamp":"2021-02-08T07:18:21.702503042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6qX65q9Zll4vnVd5kG0CkcV","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.708160_45.1","section":"bucket.instance","name":"lhddgo-2:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2","timestamp":"2021-02-08T07:18:21.708160313Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6qX65q9Zll4vnVd5kG0CkcV","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.710147_46.1","section":"bucket","name":"lhddgo-2","timestamp":"2021-02-08T07:18:21.710147097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1DMS1mNFwKdoHY2cnmaj9tt","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.714859_47.1","section":"bucket","name":"lhddgo-2","timestamp":"2021-02-08T07:18:21.714859854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1DMS1mNFwKdoHY2cnmaj9tt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768704.880852_48.1","section":"bucket.instance","name":"lhddgo-3:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3","timestamp":"2021-02-08T07:18:24.880852494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rOLGVtTLt3ajM6sppBkkd3e","ver":1},"status":{"status":"write"}}},{"id":"1_1612768704.886356_49.1","section":"bucket.instance","name":"lhddgo-3:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3","timestamp":"2021-02-08T07:18:24.886356317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rOLGVtTLt3ajM6sppBkkd3e","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768704.888224_50.1","section":"bucket","name":"lhddgo-3","timestamp":"2021-02-08T07:18:24.888224742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqx6QY3WLExCkHpK84fAXxr","ver":1},"status":{"status":"write"}}},{"id":"1_1612768704.893108_51.1","section":"bucket","name":"lhddgo-3","timestamp":"2021-02-08T07:18:24.893108028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqx6QY3WLExCkHpK84fAXxr","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.777993_52.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4","timestamp":"2021-02-08T07:18:32.777993072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BoYtQApnX7_2AtGejxpYaZO","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.783478_53.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4","timestamp":"2021-02-08T07:18:32.783478935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BoYtQApnX7_2AtGejxpYaZO","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.785319_54.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:32.785319651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fp7ohDPD_24JfOiBzo-rYXf","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.791923_55.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:32.791923400Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fp7ohDPD_24JfOiBzo-rYXf","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.814845_56.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5","timestamp":"2021-02-08T07:18:32.814845373Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4CPxPuDJA5f6reOzSejIeq","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.820494_57.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5","timestamp":"2021-02-08T07:18:32.820494612Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4CPxPuDJA5f6reOzSejIeq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.822303_58.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:32.822303238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_yHBtAEVD0Gyl9lvsZLfTL","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.830478_59.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:32.830478754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_yHBtAEVD0Gyl9lvsZLfTL","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.875274_60.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6","timestamp":"2021-02-08T07:18:32.875274066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIAw4BnGrFg8-fkZKIn5AkT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.883030_61.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6","timestamp":"2021-02-08T07:18:32.883030645Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIAw4BnGrFg8-fkZKIn5AkT","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.885168_62.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:32.885168872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hS--GkAzPDMbDD0Ah6ChxD9","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.891808_63.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:32.891808199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hS--GkAzPDMbDD0Ah6ChxD9","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.455112_64.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.7","timestamp":"2021-02-08T07:18:40.455112453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D64Kh1AQ6YPfhYh6ce23ln9","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.461143_65.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.7","timestamp":"2021-02-08T07:18:40.461143164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D64Kh1AQ6YPfhYh6ce23ln9","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.463040_66.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.463040385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1HoTcP5PU2Kgj55NSIKnwiu","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.467993_67.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.467993630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1HoTcP5PU2Kgj55NSIKnwiu","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.499332_68.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.8","timestamp":"2021-02-08T07:18:40.499332959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ghr6TovGAVCrhaoBiwCDcI0","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.504921_69.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.8","timestamp":"2021-02-08T07:18:40.504921848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ghr6TovGAVCrhaoBiwCDcI0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.507224_70.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.507224631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRWiK6cMflxrN6xErwMtL8C","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.511355_71.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.511355956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRWiK6cMflxrN6xErwMtL8C","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.543636_72.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.9","timestamp":"2021-02-08T07:18:40.543636213Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xlBzXd-b99YpCwUbXyq6T_I","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.549664_73.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.9","timestamp":"2021-02-08T07:18:40.549664095Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xlBzXd-b99YpCwUbXyq6T_I","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.551977_74.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.551977841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2tIbTzLnZNEgqsoMi4Kyu4g","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.556906_75.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.556906173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2tIbTzLnZNEgqsoMi4Kyu4g","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.598520_76.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.10","timestamp":"2021-02-08T07:18:40.598520764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_erUTYNTZcG6PubXCSk30xX6","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.603500_77.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.10","timestamp":"2021-02-08T07:18:40.603500331Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_erUTYNTZcG6PubXCSk30xX6","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.605348_78.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.605348512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3KtUz4rocFNL2_vS5Rfub48","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.608562_79.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.608562308Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3KtUz4rocFNL2_vS5Rfub48","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.654740_80.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.11","timestamp":"2021-02-08T07:18:40.654740480Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HT1xmRKztbydlE5i670276O","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.660209_81.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.11","timestamp":"2021-02-08T07:18:40.660209900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HT1xmRKztbydlE5i670276O","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.662252_82.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.662252850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7sCGVKD50Piys06LRi5mRnA","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.667287_83.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.667287114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7sCGVKD50Piys06LRi5mRnA","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.764760_84.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.12","timestamp":"2021-02-08T07:18:40.764760372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vhvVK_HjSPWSrSoiQ3B1sWk","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.770576_85.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.12","timestamp":"2021-02-08T07:18:40.770576911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vhvVK_HjSPWSrSoiQ3B1sWk","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.772523_86.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.772523598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2kLGuYGaCkV1AwIhj2Cdj0z","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.776664_87.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.776664142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2kLGuYGaCkV1AwIhj2Cdj0z","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.875434_88.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.13","timestamp":"2021-02-08T07:18:40.875434170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_szU7g2F8bRR_qjRFAsjD8UV","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.880535_89.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.13","timestamp":"2021-02-08T07:18:40.880535006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_szU7g2F8bRR_qjRFAsjD8UV","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.882328_90.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.882328424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N9j0fBH4RG41rW2px2Kfg52","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.886694_91.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.886694720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N9j0fBH4RG41rW2px2Kfg52","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.942414_92.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.14","timestamp":"2021-02-08T07:18:40.942414834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c2MoBLNAFcZhCc0-6FkAure","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.947900_93.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.14","timestamp":"2021-02-08T07:18:40.947900056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c2MoBLNAFcZhCc0-6FkAure","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.950597_94.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.950597043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o5ykWPY7kYS85AigxFS37fZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.955189_95.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.955189582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o5ykWPY7kYS85AigxFS37fZ","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768721.049288_96.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.15","timestamp":"2021-02-08T07:18:41.049288715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KNldoUM05NwRop7PdH1FruD","ver":1},"status":{"status":"write"}}},{"id":"1_1612768721.055941_97.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.15","timestamp":"2021-02-08T07:18:41.055941239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KNldoUM05NwRop7PdH1FruD","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768721.058596_98.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:41.058596909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Zk1CJRJYFjyBIHzf8tu1OT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768721.063275_99.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:41.063275680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Zk1CJRJYFjyBIHzf8tu1OT","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768729.345437_100.1","section":"bucket.instance","name":"lhddgo-7:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16","timestamp":"2021-02-08T07:18:49.345437338Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_00nF2x8tkdYhe8Yvf2J_kXR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.351031_101.1","section":"bucket.instance","name":"lhddgo-7:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16","timestamp":"2021-02-08T07:18:49.351031227Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_00nF2x8tkdYhe8Yvf2J_kXR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.353169_102.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:49.353169069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.358754_103.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:49.358754345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.382909_104.1","section":"bucket.instance","name":"lhddgo-8:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17","timestamp":"2021-02-08T07:18:49.382909424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A2V06k6mnYpoHdwsmFO6I2c","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.389686_105.1","section":"bucket.instance","name":"lhddgo-8:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17","timestamp":"2021-02-08T07:18:49.389686951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A2V06k6mnYpoHdwsmFO6I2c","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.392255_106.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:49.392255869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.398757_107.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:49.398757052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.447313_108.1","section":"bucket.instance","name":"lhddgo-9:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18","timestamp":"2021-02-08T07:18:49.447313935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dG0OIO3lIHMoZH43A6aVRSG","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.453059_109.1","section":"bucket.instance","name":"lhddgo-9:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18","timestamp":"2021-02-08T07:18:49.453059961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dG0OIO3lIHMoZH43A6aVRSG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.455135_110.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:49.455135230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.462500_111.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:49.462500549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768738.338507_112.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:58.338507156Z","data":{"read_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768738.346128_113.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:58.346128189Z","data":{"read_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768738.734584_114.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:58.734584501Z","data":{"read_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768738.742706_115.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:58.742706390Z","data":{"read_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768739.744978_116.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:59.744978355Z","data":{"read_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768739.752484_117.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:59.752484319Z","data":{"read_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768746.954486_46.1","section":"bucket.instance","name":"lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19","timestamp":"2021-02-08T07:19:06.954486618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZ2_foaAGbx_i2Sm3CpW_Kk","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.960028_55.1","section":"bucket.instance","name":"lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19","timestamp":"2021-02-08T07:19:06.960028764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZ2_foaAGbx_i2Sm3CpW_Kk","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.961876_56.1","section":"bucket","name":"lhddgo-10","timestamp":"2021-02-08T07:19:06.961876770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PahWfqgiX7tHFTo7PfKIGym","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.966814_57.1","section":"bucket","name":"lhddgo-10","timestamp":"2021-02-08T07:19:06.966814969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PahWfqgiX7tHFTo7PfKIGym","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.988241_58.1","section":"bucket.instance","name":"lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20","timestamp":"2021-02-08T07:19:06.988241783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zC5o4T_ukEDV-6O241V0z4P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.993594_59.1","section":"bucket.instance","name":"lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20","timestamp":"2021-02-08T07:19:06.993594301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zC5o4T_ukEDV-6O241V0z4P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.995614_60.1","section":"bucket","name":"lhddgo-11","timestamp":"2021-02-08T07:19:06.995614191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lkJAfr_qEGLVNdY3mgxQHeC","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.000743_61.1","section":"bucket","name":"lhddgo-11","timestamp":"2021-02-08T07:19:07.000743900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lkJAfr_qEGLVNdY3mgxQHeC","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768747.045230_63.1","section":"bucket.instance","name":"lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21","timestamp":"2021-02-08T07:19:07.045230732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7eGXe-HdPcO7HYGGIubcFdo","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.050420_64.1","section":"bucket.instance","name":"lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21","timestamp":"2021-02-08T07:19:07.050420217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7eGXe-HdPcO7HYGGIubcFdo","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768747.052392_65.1","section":"bucket","name":"lhddgo-12","timestamp":"2021-02-08T07:19:07.052392440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_vX8F929OWDaiEf5eFNP9P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.057366_66.1","section":"bucket","name":"lhddgo-12","timestamp":"2021-02-08T07:19:07.057366611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_vX8F929OWDaiEf5eFNP9P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.456382_133.1","section":"bucket.instance","name":"lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22","timestamp":"2021-02-08T07:19:23.456382197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ttLcbiaMaOBic5ZoV20Gusq","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.462897_135.1","section":"bucket.instance","name":"lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22","timestamp":"2021-02-08T07:19:23.462897145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ttLcbiaMaOBic5ZoV20Gusq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.465037_136.1","section":"bucket","name":"lhddgo-13","timestamp":"2021-02-08T07:19:23.465037077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s0fnDUyDPo0ZxNuB6AgyP4P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.471695_137.1","section":"bucket","name":"lhddgo-13","timestamp":"2021-02-08T07:19:23.471695624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s0fnDUyDPo0ZxNuB6AgyP4P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.497377_138.1","section":"bucket.instance","name":"lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23","timestamp":"2021-02-08T07:19:23.497377314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f25x5uML91RHGgZY3mqvkiD","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.503198_139.1","section":"bucket.instance","name":"lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23","timestamp":"2021-02-08T07:19:23.503198102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f25x5uML91RHGgZY3mqvkiD","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.505240_140.1","section":"bucket","name":"lhddgo-14","timestamp":"2021-02-08T07:19:23.505240280Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yfbtCQsCHEWsWQtRDJWU0TK","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.510426_141.1","section":"bucket","name":"lhddgo-14","timestamp":"2021-02-08T07:19:23.510426290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yfbtCQsCHEWsWQtRDJWU0TK","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.556298_142.1","section":"bucket.instance","name":"lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24","timestamp":"2021-02-08T07:19:23.556298278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_86ctNhKnzUyhf-uA2VFMTRR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.563119_143.1","section":"bucket.instance","name":"lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24","timestamp":"2021-02-08T07:19:23.563119127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_86ctNhKnzUyhf-uA2VFMTRR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.565115_144.1","section":"bucket","name":"lhddgo-15","timestamp":"2021-02-08T07:19:23.565115398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kxIgLsiJU_eIuv-WaTRYw-R","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.570467_145.1","section":"bucket","name":"lhddgo-15","timestamp":"2021-02-08T07:19:23.570467263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kxIgLsiJU_eIuv-WaTRYw-R","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.007216_146.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.007216510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.012745_159.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.012745436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.014570_160.1","section":"bucket","name":"lhddgo-16","timestamp":"2021-02-08T07:20:12.014570514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nekdSdSgiiFVOM3sLjoHrs0","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.020148_161.1","section":"bucket","name":"lhddgo-16","timestamp":"2021-02-08T07:20:12.020148921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nekdSdSgiiFVOM3sLjoHrs0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.045071_162.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.045071037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.050744_163.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.050744782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.052634_164.1","section":"bucket","name":"lhddgo-17","timestamp":"2021-02-08T07:20:12.052634245Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__7j--ODbV4i8iflCZWp9f8W","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.057766_165.1","section":"bucket","name":"lhddgo-17","timestamp":"2021-02-08T07:20:12.057766474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__7j--ODbV4i8iflCZWp9f8W","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.102645_166.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.102645691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.108722_167.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.108722889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.110763_168.1","section":"bucket","name":"lhddgo-18","timestamp":"2021-02-08T07:20:12.110763801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zK0nzDVJpyF9AMlQzKt7CIi","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.116166_169.1","section":"bucket","name":"lhddgo-18","timestamp":"2021-02-08T07:20:12.116166136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zK0nzDVJpyF9AMlQzKt7CIi","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.139872_170.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.139872515Z","data":{"read_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.145372_171.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.145372779Z","data":{"read_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768812.149764_172.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.149764590Z","data":{"read_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.155292_173.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.155292011Z","data":{"read_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768812.165468_174.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.165468554Z","data":{"read_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.172314_175.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.172314820Z","data":{"read_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768909.754047_176.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.754047254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"status":{"status":"write"}}},{"id":"1_1612768909.759899_215.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.759899359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768909.761790_216.1","section":"bucket","name":"lhddgo-19","timestamp":"2021-02-08T07:21:49.761790823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7UU_grw1aDKDg8-iLh2ozDM","ver":1},"status":{"status":"write"}}},{"id":"1_1612768909.766466_217.1","section":"bucket","name":"lhddgo-19","timestamp":"2021-02-08T07:21:49.766466920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7UU_grw1aDKDg8-iLh2ozDM","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768909.772573_218.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.772573230Z","data":{"read_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":2},"status":{"status":"write"}}},{"id":"1_1612768909.778782_219.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.778782168Z","data":{"read_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768974.725779_233.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:22:54.725779997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"status":{"status":"write"}}},{"id":"1_1612768974.732118_239.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:22:54.732118058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768974.734076_240.1","section":"bucket","name":"lhddgo-20","timestamp":"2021-02-08T07:22:54.734076231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ewPFRqRbQtYiKfY1emF_0kh","ver":1},"status":{"status":"write"}}},{"id":"1_1612768974.739752_241.1","section":"bucket","name":"lhddgo-20","timestamp":"2021-02-08T07:22:54.739752084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ewPFRqRbQtYiKfY1emF_0kh","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769001.637403_242.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:21.637403627Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"status":{"status":"write"}}},{"id":"1_1612769001.644747_249.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:21.644747197Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769021.196864_250.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:41.196864849Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":3},"status":{"status":"write"}}},{"id":"1_1612769021.207297_257.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:41.207297516Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769042.799574_258.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.799574420Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.805549_265.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.805549492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.807942_266.1","section":"bucket","name":"lhddgo-21","timestamp":"2021-02-08T07:24:02.807942128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-4b49qybZ0UkKdoSGfItB4t","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.813228_267.1","section":"bucket","name":"lhddgo-21","timestamp":"2021-02-08T07:24:02.813228410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-4b49qybZ0UkKdoSGfItB4t","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.838690_268.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.838690572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.844449_269.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.844449108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.846408_270.1","section":"bucket","name":"lhddgo-22","timestamp":"2021-02-08T07:24:02.846408624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jUIV-7to8G5ZTs1obxvJjJG","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.851416_271.1","section":"bucket","name":"lhddgo-22","timestamp":"2021-02-08T07:24:02.851416259Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jUIV-7to8G5ZTs1obxvJjJG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.895189_272.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.895189936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.900758_273.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.900758211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.902745_274.1","section":"bucket","name":"lhddgo-23","timestamp":"2021-02-08T07:24:02.902745272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OjI4iZOJTWFwklM66d8DZiZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.907816_275.1","section":"bucket","name":"lhddgo-23","timestamp":"2021-02-08T07:24:02.907816855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OjI4iZOJTWFwklM66d8DZiZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.934689_276.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.934689328Z","data":{"read_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.942454_277.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.942454508Z","data":{"read_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769042.948058_278.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.948058553Z","data":{"read_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.956522_279.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.956522086Z","data":{"read_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769042.968947_280.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.968947604Z","data":{"read_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.978250_281.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.978250456Z","data":{"read_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.623875_282.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.623875923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.630835_299.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.630835843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.633069_300.1","section":"bucket","name":"lhddgo-24","timestamp":"2021-02-08T07:24:29.633069001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4t2fafA9W7emjovORdu4vzB","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.637883_301.1","section":"bucket","name":"lhddgo-24","timestamp":"2021-02-08T07:24:29.637883250Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4t2fafA9W7emjovORdu4vzB","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.660183_302.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.660183105Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.665900_303.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.665900158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.667859_304.1","section":"bucket","name":"lhddgo-25","timestamp":"2021-02-08T07:24:29.667859791Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5cBMeEkIR40rErwMHaeqFJ2","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.673799_305.1","section":"bucket","name":"lhddgo-25","timestamp":"2021-02-08T07:24:29.673799031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5cBMeEkIR40rErwMHaeqFJ2","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.720508_306.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.720508838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.726120_307.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.726120869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.727996_308.1","section":"bucket","name":"lhddgo-26","timestamp":"2021-02-08T07:24:29.727996128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__bV0SY4m7IJ85i318hYa99f","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.732935_309.1","section":"bucket","name":"lhddgo-26","timestamp":"2021-02-08T07:24:29.732935776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__bV0SY4m7IJ85i318hYa99f","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.767812_310.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.767812812Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.774076_311.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.774076941Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.778099_312.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.778099238Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.785267_313.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.785267130Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769069.790204_314.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.790204719Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.797562_315.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.797562958Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.809280_316.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.809280114Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.816992_317.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.816992209Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769069.828918_318.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.828918200Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.837623_319.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.837623812Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.849480_320.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.849480608Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.857413_321.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.857413152Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769098.920309_322.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:58.920309067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.927047_339.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:58.927047541Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.929208_340.1","section":"bucket","name":"lhddgo-27","timestamp":"2021-02-08T07:24:58.929208268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E3g7A0Syvz7sJwkowawNoEA","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.934262_341.1","section":"bucket","name":"lhddgo-27","timestamp":"2021-02-08T07:24:58.934262099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E3g7A0Syvz7sJwkowawNoEA","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.961964_342.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:58.961964655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.968667_343.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:58.968667208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.970843_344.1","section":"bucket","name":"lhddgo-28","timestamp":"2021-02-08T07:24:58.970843625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yXVDMAYob-zCi8EUwg1b8rK","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.976100_345.1","section":"bucket","name":"lhddgo-28","timestamp":"2021-02-08T07:24:58.976100609Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yXVDMAYob-zCi8EUwg1b8rK","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.021389_346.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.021389397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.027116_347.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.027116452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.029204_348.1","section":"bucket","name":"lhddgo-29","timestamp":"2021-02-08T07:24:59.029204248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BBdXyLdhOQBxsIZPgiWK1Io","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.040870_349.1","section":"bucket","name":"lhddgo-29","timestamp":"2021-02-08T07:24:59.040870109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BBdXyLdhOQBxsIZPgiWK1Io","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.070082_350.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:59.070082042Z","data":{"read_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.076291_351.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:59.076291919Z","data":{"read_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.128145_352.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:59.128145356Z","data":{"read_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.138654_353.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:59.138654984Z","data":{"read_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.195524_354.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.195524900Z","data":{"read_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.205763_355.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.205763967Z","data":{"read_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.283799_356.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.283799364Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.289172_357.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.289172513Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.291276_358.1","section":"bucket","name":"lhddgo-30","timestamp":"2021-02-08T07:24:59.291276569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UwjaMGoy_DrUnxaidzTazZd","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.296560_359.1","section":"bucket","name":"lhddgo-30","timestamp":"2021-02-08T07:24:59.296560718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UwjaMGoy_DrUnxaidzTazZd","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.322298_360.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.322298131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.328870_361.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.328870951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.331060_362.1","section":"bucket","name":"lhddgo-31","timestamp":"2021-02-08T07:24:59.331060564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2QZd7jhCZ6WbQQApzYZ6Zuq","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.336978_363.1","section":"bucket","name":"lhddgo-31","timestamp":"2021-02-08T07:24:59.336978349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2QZd7jhCZ6WbQQApzYZ6Zuq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.384623_364.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.384623961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.390826_365.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.390826118Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.392754_366.1","section":"bucket","name":"lhddgo-32","timestamp":"2021-02-08T07:24:59.392754749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bqm4fq2xDjXcNewXd8Ho5au","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.397952_367.1","section":"bucket","name":"lhddgo-32","timestamp":"2021-02-08T07:24:59.397952631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bqm4fq2xDjXcNewXd8Ho5au","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.425432_368.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.425432089Z","data":{"read_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.431859_369.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.431859354Z","data":{"read_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.481778_370.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.481778111Z","data":{"read_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.492185_371.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.492185468Z","data":{"read_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.548697_372.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.548697908Z","data":{"read_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.554553_373.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.554553520Z","data":{"read_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.625575_374.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.625575517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.632352_375.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.632352418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.634874_376.1","section":"bucket","name":"lhddgo-33","timestamp":"2021-02-08T07:24:59.634874165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S1nyQKgyKti5lpNsSFRpOE5","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.640734_377.1","section":"bucket","name":"lhddgo-33","timestamp":"2021-02-08T07:24:59.640734750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S1nyQKgyKti5lpNsSFRpOE5","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.671312_378.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:24:59.671312553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.679437_379.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:24:59.679437197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.681736_380.1","section":"bucket","name":"lhddgo-34","timestamp":"2021-02-08T07:24:59.681736519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BeJshNgmDKzGicFSITjzLQF","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.686854_381.1","section":"bucket","name":"lhddgo-34","timestamp":"2021-02-08T07:24:59.686854680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BeJshNgmDKzGicFSITjzLQF","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.734811_382.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:24:59.734811285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.740722_383.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:24:59.740722499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.742608_384.1","section":"bucket","name":"lhddgo-35","timestamp":"2021-02-08T07:24:59.742608646Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q-9lHc8CrdYvDkHWoFJqG8X","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.748237_385.1","section":"bucket","name":"lhddgo-35","timestamp":"2021-02-08T07:24:59.748237598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q-9lHc8CrdYvDkHWoFJqG8X","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.776781_386.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.776781310Z","data":{"read_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.783360_387.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.783360110Z","data":{"read_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769100.836953_388.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:25:00.836953168Z","data":{"read_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":2},"status":{"status":"write"}}},{"id":"1_1612769100.848578_389.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:25:00.848578115Z","data":{"read_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769102.144127_390.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:25:02.144127048Z","data":{"read_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":2},"status":{"status":"write"}}},{"id":"1_1612769102.156036_391.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:25:02.156036210Z","data":{"read_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769102.760731_392.1","section":"bucket.instance","name":"lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45","timestamp":"2021-02-08T07:25:02.760731916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pUmMyR0Isdm2KnONKSzA-Ey","ver":1},"status":{"status":"write"}}},{"id":"1_1612769102.767819_393.1","section":"bucket.instance","name":"lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45","timestamp":"2021-02-08T07:25:02.767819540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pUmMyR0Isdm2KnONKSzA-Ey","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769102.770229_394.1","section":"bucket","name":"lhddgo-36","timestamp":"2021-02-08T07:25:02.770229312Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8bkeVL24JNAUI6gG0schLP","ver":1},"status":{"status":"write"}}},{"id":"1_1612769102.775801_395.1","section":"bucket","name":"lhddgo-36","timestamp":"2021-02-08T07:25:02.775801188Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8bkeVL24JNAUI6gG0schLP","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.117151_396.1","section":"bucket.instance","name":"lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46","timestamp":"2021-02-08T07:25:03.117151764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g5Dq8aX87ZwjcUVPxfbcfaL","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.122644_397.1","section":"bucket.instance","name":"lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46","timestamp":"2021-02-08T07:25:03.122644962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g5Dq8aX87ZwjcUVPxfbcfaL","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.124978_398.1","section":"bucket","name":"lhddgo-37","timestamp":"2021-02-08T07:25:03.124978224Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__2oxvZ6RrkXDCfhasSdxBZQ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.130437_399.1","section":"bucket","name":"lhddgo-37","timestamp":"2021-02-08T07:25:03.130437751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__2oxvZ6RrkXDCfhasSdxBZQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.182592_400.1","section":"bucket.instance","name":"lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47","timestamp":"2021-02-08T07:25:03.182592377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_06zd8oUyALlG1ZsQD8Fdol-","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.187922_401.1","section":"bucket.instance","name":"lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47","timestamp":"2021-02-08T07:25:03.187922879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_06zd8oUyALlG1ZsQD8Fdol-","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.189848_402.1","section":"bucket","name":"lhddgo-38","timestamp":"2021-02-08T07:25:03.189848323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUOpEUVHAB91BG5u5pJNnrO","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.194698_403.1","section":"bucket","name":"lhddgo-38","timestamp":"2021-02-08T07:25:03.194698092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUOpEUVHAB91BG5u5pJNnrO","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769106.694646_404.1","section":"bucket.instance","name":"lhddgo-39:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48","timestamp":"2021-02-08T07:25:06.694646856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CNAF5lCyovQyVl-qG7T3ANF","ver":1},"status":{"status":"write"}}},{"id":"1_1612769106.699588_406.1","section":"bucket.instance","name":"lhddgo-39:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48","timestamp":"2021-02-08T07:25:06.699588368Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CNAF5lCyovQyVl-qG7T3ANF","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769106.701410_407.1","section":"bucket","name":"lhddgo-39","timestamp":"2021-02-08T07:25:06.701410302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1SwIrqXTWHi7vifEqt55SsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769106.705845_408.1","section":"bucket","name":"lhddgo-39","timestamp":"2021-02-08T07:25:06.705845190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1SwIrqXTWHi7vifEqt55SsQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.036501_409.1","section":"bucket.instance","name":"lhddgo-40:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49","timestamp":"2021-02-08T07:25:07.036501974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dDggL3R9PYxxraDauffrO09","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.042000_410.1","section":"bucket.instance","name":"lhddgo-40:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49","timestamp":"2021-02-08T07:25:07.042000544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dDggL3R9PYxxraDauffrO09","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.043987_411.1","section":"bucket","name":"lhddgo-40","timestamp":"2021-02-08T07:25:07.043987355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZOZ_VlHozV1b8Q-DrynRt61","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.049028_412.1","section":"bucket","name":"lhddgo-40","timestamp":"2021-02-08T07:25:07.049028688Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZOZ_VlHozV1b8Q-DrynRt61","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.097668_413.1","section":"bucket.instance","name":"lhddgo-41:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50","timestamp":"2021-02-08T07:25:07.097668057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTrt6zeu476exuMMzu3hcGZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.103981_414.1","section":"bucket.instance","name":"lhddgo-41:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50","timestamp":"2021-02-08T07:25:07.103981325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTrt6zeu476exuMMzu3hcGZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.106208_415.1","section":"bucket","name":"lhddgo-41","timestamp":"2021-02-08T07:25:07.106208500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YtG89TnLF4vYQMJMWzkgy3J","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.111803_416.1","section":"bucket","name":"lhddgo-41","timestamp":"2021-02-08T07:25:07.111803603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YtG89TnLF4vYQMJMWzkgy3J","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:41.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:41.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:41.637 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period d4b11cd5-1a7c-4383-afcf-402844a1791d --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:41.674 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:41.674 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:41.921 INFO:teuthology.orchestra.run.smithi110.stdout:[{"id":"1_1612768579.522956_30.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-08T07:16:19.522956903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GTmvG-EF-9cF89V4myGNcR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768579.532632_32.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-08T07:16:19.532632715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GTmvG-EF-9cF89V4myGNcR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.649968_40.1","section":"bucket.instance","name":"lhddgo-1:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1","timestamp":"2021-02-08T07:18:21.649968470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e2Im8Aginrg7vTQ5Uq0UExs","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.675446_41.1","section":"bucket.instance","name":"lhddgo-1:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1","timestamp":"2021-02-08T07:18:21.675446655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e2Im8Aginrg7vTQ5Uq0UExs","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.677384_42.1","section":"bucket","name":"lhddgo-1","timestamp":"2021-02-08T07:18:21.677384640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m_Fn4BKOiM5g7mlQklEvF8h","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.681366_43.1","section":"bucket","name":"lhddgo-1","timestamp":"2021-02-08T07:18:21.681366592Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m_Fn4BKOiM5g7mlQklEvF8h","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.702503_44.1","section":"bucket.instance","name":"lhddgo-2:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2","timestamp":"2021-02-08T07:18:21.702503042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6qX65q9Zll4vnVd5kG0CkcV","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.708160_45.1","section":"bucket.instance","name":"lhddgo-2:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2","timestamp":"2021-02-08T07:18:21.708160313Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6qX65q9Zll4vnVd5kG0CkcV","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768701.710147_46.1","section":"bucket","name":"lhddgo-2","timestamp":"2021-02-08T07:18:21.710147097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1DMS1mNFwKdoHY2cnmaj9tt","ver":1},"status":{"status":"write"}}},{"id":"1_1612768701.714859_47.1","section":"bucket","name":"lhddgo-2","timestamp":"2021-02-08T07:18:21.714859854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1DMS1mNFwKdoHY2cnmaj9tt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768704.880852_48.1","section":"bucket.instance","name":"lhddgo-3:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3","timestamp":"2021-02-08T07:18:24.880852494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rOLGVtTLt3ajM6sppBkkd3e","ver":1},"status":{"status":"write"}}},{"id":"1_1612768704.886356_49.1","section":"bucket.instance","name":"lhddgo-3:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3","timestamp":"2021-02-08T07:18:24.886356317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rOLGVtTLt3ajM6sppBkkd3e","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768704.888224_50.1","section":"bucket","name":"lhddgo-3","timestamp":"2021-02-08T07:18:24.888224742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqx6QY3WLExCkHpK84fAXxr","ver":1},"status":{"status":"write"}}},{"id":"1_1612768704.893108_51.1","section":"bucket","name":"lhddgo-3","timestamp":"2021-02-08T07:18:24.893108028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqx6QY3WLExCkHpK84fAXxr","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.777993_52.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4","timestamp":"2021-02-08T07:18:32.777993072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BoYtQApnX7_2AtGejxpYaZO","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.783478_53.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4","timestamp":"2021-02-08T07:18:32.783478935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BoYtQApnX7_2AtGejxpYaZO","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.785319_54.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:32.785319651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fp7ohDPD_24JfOiBzo-rYXf","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.791923_55.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:32.791923400Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fp7ohDPD_24JfOiBzo-rYXf","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.814845_56.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5","timestamp":"2021-02-08T07:18:32.814845373Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4CPxPuDJA5f6reOzSejIeq","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.820494_57.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5","timestamp":"2021-02-08T07:18:32.820494612Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4CPxPuDJA5f6reOzSejIeq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.822303_58.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:32.822303238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_yHBtAEVD0Gyl9lvsZLfTL","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.830478_59.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:32.830478754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_yHBtAEVD0Gyl9lvsZLfTL","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.875274_60.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6","timestamp":"2021-02-08T07:18:32.875274066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIAw4BnGrFg8-fkZKIn5AkT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.883030_61.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6","timestamp":"2021-02-08T07:18:32.883030645Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIAw4BnGrFg8-fkZKIn5AkT","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768712.885168_62.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:32.885168872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hS--GkAzPDMbDD0Ah6ChxD9","ver":1},"status":{"status":"write"}}},{"id":"1_1612768712.891808_63.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:32.891808199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hS--GkAzPDMbDD0Ah6ChxD9","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.455112_64.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.7","timestamp":"2021-02-08T07:18:40.455112453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D64Kh1AQ6YPfhYh6ce23ln9","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.461143_65.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.7","timestamp":"2021-02-08T07:18:40.461143164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D64Kh1AQ6YPfhYh6ce23ln9","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.463040_66.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.463040385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1HoTcP5PU2Kgj55NSIKnwiu","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.467993_67.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.467993630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1HoTcP5PU2Kgj55NSIKnwiu","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.499332_68.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.8","timestamp":"2021-02-08T07:18:40.499332959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ghr6TovGAVCrhaoBiwCDcI0","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.504921_69.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.8","timestamp":"2021-02-08T07:18:40.504921848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ghr6TovGAVCrhaoBiwCDcI0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.507224_70.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.507224631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRWiK6cMflxrN6xErwMtL8C","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.511355_71.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.511355956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRWiK6cMflxrN6xErwMtL8C","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.543636_72.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.9","timestamp":"2021-02-08T07:18:40.543636213Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xlBzXd-b99YpCwUbXyq6T_I","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.549664_73.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.9","timestamp":"2021-02-08T07:18:40.549664095Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xlBzXd-b99YpCwUbXyq6T_I","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.551977_74.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.551977841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2tIbTzLnZNEgqsoMi4Kyu4g","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.556906_75.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.556906173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2tIbTzLnZNEgqsoMi4Kyu4g","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.598520_76.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.10","timestamp":"2021-02-08T07:18:40.598520764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_erUTYNTZcG6PubXCSk30xX6","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.603500_77.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.10","timestamp":"2021-02-08T07:18:40.603500331Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_erUTYNTZcG6PubXCSk30xX6","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.605348_78.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.605348512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3KtUz4rocFNL2_vS5Rfub48","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.608562_79.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.608562308Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3KtUz4rocFNL2_vS5Rfub48","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.654740_80.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.11","timestamp":"2021-02-08T07:18:40.654740480Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HT1xmRKztbydlE5i670276O","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.660209_81.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.11","timestamp":"2021-02-08T07:18:40.660209900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HT1xmRKztbydlE5i670276O","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.662252_82.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.662252850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7sCGVKD50Piys06LRi5mRnA","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.667287_83.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.667287114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7sCGVKD50Piys06LRi5mRnA","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.764760_84.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.12","timestamp":"2021-02-08T07:18:40.764760372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vhvVK_HjSPWSrSoiQ3B1sWk","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.770576_85.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.12","timestamp":"2021-02-08T07:18:40.770576911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vhvVK_HjSPWSrSoiQ3B1sWk","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.772523_86.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.772523598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2kLGuYGaCkV1AwIhj2Cdj0z","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.776664_87.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:40.776664142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2kLGuYGaCkV1AwIhj2Cdj0z","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.875434_88.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.13","timestamp":"2021-02-08T07:18:40.875434170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_szU7g2F8bRR_qjRFAsjD8UV","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.880535_89.1","section":"bucket.instance","name":"lhddgo-4:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.13","timestamp":"2021-02-08T07:18:40.880535006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_szU7g2F8bRR_qjRFAsjD8UV","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.882328_90.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.882328424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N9j0fBH4RG41rW2px2Kfg52","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.886694_91.1","section":"bucket","name":"lhddgo-4","timestamp":"2021-02-08T07:18:40.886694720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N9j0fBH4RG41rW2px2Kfg52","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768720.942414_92.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.14","timestamp":"2021-02-08T07:18:40.942414834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c2MoBLNAFcZhCc0-6FkAure","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.947900_93.1","section":"bucket.instance","name":"lhddgo-5:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.14","timestamp":"2021-02-08T07:18:40.947900056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c2MoBLNAFcZhCc0-6FkAure","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768720.950597_94.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.950597043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o5ykWPY7kYS85AigxFS37fZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612768720.955189_95.1","section":"bucket","name":"lhddgo-5","timestamp":"2021-02-08T07:18:40.955189582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o5ykWPY7kYS85AigxFS37fZ","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768721.049288_96.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.15","timestamp":"2021-02-08T07:18:41.049288715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KNldoUM05NwRop7PdH1FruD","ver":1},"status":{"status":"write"}}},{"id":"1_1612768721.055941_97.1","section":"bucket.instance","name":"lhddgo-6:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.15","timestamp":"2021-02-08T07:18:41.055941239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KNldoUM05NwRop7PdH1FruD","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768721.058596_98.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:41.058596909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Zk1CJRJYFjyBIHzf8tu1OT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768721.063275_99.1","section":"bucket","name":"lhddgo-6","timestamp":"2021-02-08T07:18:41.063275680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Zk1CJRJYFjyBIHzf8tu1OT","ver":1},"status":{"status":"abort"}}},{"id":"1_1612768729.345437_100.1","section":"bucket.instance","name":"lhddgo-7:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16","timestamp":"2021-02-08T07:18:49.345437338Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_00nF2x8tkdYhe8Yvf2J_kXR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.351031_101.1","section":"bucket.instance","name":"lhddgo-7:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16","timestamp":"2021-02-08T07:18:49.351031227Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_00nF2x8tkdYhe8Yvf2J_kXR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.353169_102.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:49.353169069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.358754_103.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:49.358754345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.382909_104.1","section":"bucket.instance","name":"lhddgo-8:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17","timestamp":"2021-02-08T07:18:49.382909424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A2V06k6mnYpoHdwsmFO6I2c","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.389686_105.1","section":"bucket.instance","name":"lhddgo-8:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17","timestamp":"2021-02-08T07:18:49.389686951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A2V06k6mnYpoHdwsmFO6I2c","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.392255_106.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:49.392255869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.398757_107.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:49.398757052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.447313_108.1","section":"bucket.instance","name":"lhddgo-9:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18","timestamp":"2021-02-08T07:18:49.447313935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dG0OIO3lIHMoZH43A6aVRSG","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.453059_109.1","section":"bucket.instance","name":"lhddgo-9:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18","timestamp":"2021-02-08T07:18:49.453059961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dG0OIO3lIHMoZH43A6aVRSG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768729.455135_110.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:49.455135230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"status":{"status":"write"}}},{"id":"1_1612768729.462500_111.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:49.462500549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768738.338507_112.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:58.338507156Z","data":{"read_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768738.346128_113.1","section":"bucket","name":"lhddgo-7","timestamp":"2021-02-08T07:18:58.346128189Z","data":{"read_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":1},"write_version":{"tag":"_4BxK2D1Q6dCARR_9hzqEt6H","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768738.734584_114.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:58.734584501Z","data":{"read_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768738.742706_115.1","section":"bucket","name":"lhddgo-8","timestamp":"2021-02-08T07:18:58.742706390Z","data":{"read_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":1},"write_version":{"tag":"_C303xFXdqEM3fmd4Fdz0rI_","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768739.744978_116.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:59.744978355Z","data":{"read_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":2},"status":{"status":"remove"}}},{"id":"1_1612768739.752484_117.1","section":"bucket","name":"lhddgo-9","timestamp":"2021-02-08T07:18:59.752484319Z","data":{"read_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":1},"write_version":{"tag":"_VbItIWhXnHP4YTdMIzj51uE","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768746.954486_46.1","section":"bucket.instance","name":"lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19","timestamp":"2021-02-08T07:19:06.954486618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZ2_foaAGbx_i2Sm3CpW_Kk","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.960028_55.1","section":"bucket.instance","name":"lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19","timestamp":"2021-02-08T07:19:06.960028764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZ2_foaAGbx_i2Sm3CpW_Kk","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.961876_56.1","section":"bucket","name":"lhddgo-10","timestamp":"2021-02-08T07:19:06.961876770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PahWfqgiX7tHFTo7PfKIGym","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.966814_57.1","section":"bucket","name":"lhddgo-10","timestamp":"2021-02-08T07:19:06.966814969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PahWfqgiX7tHFTo7PfKIGym","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.988241_58.1","section":"bucket.instance","name":"lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20","timestamp":"2021-02-08T07:19:06.988241783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zC5o4T_ukEDV-6O241V0z4P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768746.993594_59.1","section":"bucket.instance","name":"lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20","timestamp":"2021-02-08T07:19:06.993594301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zC5o4T_ukEDV-6O241V0z4P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768746.995614_60.1","section":"bucket","name":"lhddgo-11","timestamp":"2021-02-08T07:19:06.995614191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lkJAfr_qEGLVNdY3mgxQHeC","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.000743_61.1","section":"bucket","name":"lhddgo-11","timestamp":"2021-02-08T07:19:07.000743900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lkJAfr_qEGLVNdY3mgxQHeC","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768747.045230_63.1","section":"bucket.instance","name":"lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21","timestamp":"2021-02-08T07:19:07.045230732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7eGXe-HdPcO7HYGGIubcFdo","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.050420_64.1","section":"bucket.instance","name":"lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21","timestamp":"2021-02-08T07:19:07.050420217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7eGXe-HdPcO7HYGGIubcFdo","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768747.052392_65.1","section":"bucket","name":"lhddgo-12","timestamp":"2021-02-08T07:19:07.052392440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_vX8F929OWDaiEf5eFNP9P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768747.057366_66.1","section":"bucket","name":"lhddgo-12","timestamp":"2021-02-08T07:19:07.057366611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W_vX8F929OWDaiEf5eFNP9P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.456382_133.1","section":"bucket.instance","name":"lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22","timestamp":"2021-02-08T07:19:23.456382197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ttLcbiaMaOBic5ZoV20Gusq","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.462897_135.1","section":"bucket.instance","name":"lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22","timestamp":"2021-02-08T07:19:23.462897145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ttLcbiaMaOBic5ZoV20Gusq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.465037_136.1","section":"bucket","name":"lhddgo-13","timestamp":"2021-02-08T07:19:23.465037077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s0fnDUyDPo0ZxNuB6AgyP4P","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.471695_137.1","section":"bucket","name":"lhddgo-13","timestamp":"2021-02-08T07:19:23.471695624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s0fnDUyDPo0ZxNuB6AgyP4P","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.497377_138.1","section":"bucket.instance","name":"lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23","timestamp":"2021-02-08T07:19:23.497377314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f25x5uML91RHGgZY3mqvkiD","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.503198_139.1","section":"bucket.instance","name":"lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23","timestamp":"2021-02-08T07:19:23.503198102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f25x5uML91RHGgZY3mqvkiD","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.505240_140.1","section":"bucket","name":"lhddgo-14","timestamp":"2021-02-08T07:19:23.505240280Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yfbtCQsCHEWsWQtRDJWU0TK","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.510426_141.1","section":"bucket","name":"lhddgo-14","timestamp":"2021-02-08T07:19:23.510426290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yfbtCQsCHEWsWQtRDJWU0TK","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.556298_142.1","section":"bucket.instance","name":"lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24","timestamp":"2021-02-08T07:19:23.556298278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_86ctNhKnzUyhf-uA2VFMTRR","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.563119_143.1","section":"bucket.instance","name":"lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24","timestamp":"2021-02-08T07:19:23.563119127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_86ctNhKnzUyhf-uA2VFMTRR","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768763.565115_144.1","section":"bucket","name":"lhddgo-15","timestamp":"2021-02-08T07:19:23.565115398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kxIgLsiJU_eIuv-WaTRYw-R","ver":1},"status":{"status":"write"}}},{"id":"1_1612768763.570467_145.1","section":"bucket","name":"lhddgo-15","timestamp":"2021-02-08T07:19:23.570467263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kxIgLsiJU_eIuv-WaTRYw-R","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.007216_146.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.007216510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.012745_159.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.012745436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.014570_160.1","section":"bucket","name":"lhddgo-16","timestamp":"2021-02-08T07:20:12.014570514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nekdSdSgiiFVOM3sLjoHrs0","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.020148_161.1","section":"bucket","name":"lhddgo-16","timestamp":"2021-02-08T07:20:12.020148921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nekdSdSgiiFVOM3sLjoHrs0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.045071_162.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.045071037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.050744_163.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.050744782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.052634_164.1","section":"bucket","name":"lhddgo-17","timestamp":"2021-02-08T07:20:12.052634245Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__7j--ODbV4i8iflCZWp9f8W","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.057766_165.1","section":"bucket","name":"lhddgo-17","timestamp":"2021-02-08T07:20:12.057766474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__7j--ODbV4i8iflCZWp9f8W","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.102645_166.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.102645691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.108722_167.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.108722889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.110763_168.1","section":"bucket","name":"lhddgo-18","timestamp":"2021-02-08T07:20:12.110763801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zK0nzDVJpyF9AMlQzKt7CIi","ver":1},"status":{"status":"write"}}},{"id":"1_1612768812.116166_169.1","section":"bucket","name":"lhddgo-18","timestamp":"2021-02-08T07:20:12.116166136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zK0nzDVJpyF9AMlQzKt7CIi","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768812.139872_170.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.139872515Z","data":{"read_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.145372_171.1","section":"bucket.instance","name":"lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25","timestamp":"2021-02-08T07:20:12.145372779Z","data":{"read_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":1},"write_version":{"tag":"_PiH2dWaiwvkA9d8V4iNlSJN","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768812.149764_172.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.149764590Z","data":{"read_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.155292_173.1","section":"bucket.instance","name":"lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26","timestamp":"2021-02-08T07:20:12.155292011Z","data":{"read_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":1},"write_version":{"tag":"_UTw4DWaasTSaD4FC9pHRhmT","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768812.165468_174.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.165468554Z","data":{"read_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":2},"status":{"status":"write"}}},{"id":"1_1612768812.172314_175.1","section":"bucket.instance","name":"lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27","timestamp":"2021-02-08T07:20:12.172314820Z","data":{"read_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":1},"write_version":{"tag":"_CVWpg80Tt37K9VSBDcDU2_v","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768909.754047_176.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.754047254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"status":{"status":"write"}}},{"id":"1_1612768909.759899_215.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.759899359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768909.761790_216.1","section":"bucket","name":"lhddgo-19","timestamp":"2021-02-08T07:21:49.761790823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7UU_grw1aDKDg8-iLh2ozDM","ver":1},"status":{"status":"write"}}},{"id":"1_1612768909.766466_217.1","section":"bucket","name":"lhddgo-19","timestamp":"2021-02-08T07:21:49.766466920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7UU_grw1aDKDg8-iLh2ozDM","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768909.772573_218.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.772573230Z","data":{"read_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":2},"status":{"status":"write"}}},{"id":"1_1612768909.778782_219.1","section":"bucket.instance","name":"lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28","timestamp":"2021-02-08T07:21:49.778782168Z","data":{"read_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":1},"write_version":{"tag":"_rUpFDx_at0ZMeTY-CmzG61V","ver":2},"status":{"status":"complete"}}},{"id":"1_1612768974.725779_233.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:22:54.725779997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"status":{"status":"write"}}},{"id":"1_1612768974.732118_239.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:22:54.732118058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"status":{"status":"complete"}}},{"id":"1_1612768974.734076_240.1","section":"bucket","name":"lhddgo-20","timestamp":"2021-02-08T07:22:54.734076231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ewPFRqRbQtYiKfY1emF_0kh","ver":1},"status":{"status":"write"}}},{"id":"1_1612768974.739752_241.1","section":"bucket","name":"lhddgo-20","timestamp":"2021-02-08T07:22:54.739752084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ewPFRqRbQtYiKfY1emF_0kh","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769001.637403_242.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:21.637403627Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"status":{"status":"write"}}},{"id":"1_1612769001.644747_249.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:21.644747197Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":1},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769021.196864_250.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:41.196864849Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":3},"status":{"status":"write"}}},{"id":"1_1612769021.207297_257.1","section":"bucket.instance","name":"lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29","timestamp":"2021-02-08T07:23:41.207297516Z","data":{"read_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":2},"write_version":{"tag":"_i-xbj3GLG5Qwdn5HGXPIil6","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769042.799574_258.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.799574420Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.805549_265.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.805549492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.807942_266.1","section":"bucket","name":"lhddgo-21","timestamp":"2021-02-08T07:24:02.807942128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-4b49qybZ0UkKdoSGfItB4t","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.813228_267.1","section":"bucket","name":"lhddgo-21","timestamp":"2021-02-08T07:24:02.813228410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-4b49qybZ0UkKdoSGfItB4t","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.838690_268.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.838690572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.844449_269.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.844449108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.846408_270.1","section":"bucket","name":"lhddgo-22","timestamp":"2021-02-08T07:24:02.846408624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jUIV-7to8G5ZTs1obxvJjJG","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.851416_271.1","section":"bucket","name":"lhddgo-22","timestamp":"2021-02-08T07:24:02.851416259Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jUIV-7to8G5ZTs1obxvJjJG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.895189_272.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.895189936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.900758_273.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.900758211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.902745_274.1","section":"bucket","name":"lhddgo-23","timestamp":"2021-02-08T07:24:02.902745272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OjI4iZOJTWFwklM66d8DZiZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769042.907816_275.1","section":"bucket","name":"lhddgo-23","timestamp":"2021-02-08T07:24:02.907816855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OjI4iZOJTWFwklM66d8DZiZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769042.934689_276.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.934689328Z","data":{"read_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.942454_277.1","section":"bucket.instance","name":"lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30","timestamp":"2021-02-08T07:24:02.942454508Z","data":{"read_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":1},"write_version":{"tag":"_TdwzgVoSEVFYSgi-Lk99em7","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769042.948058_278.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.948058553Z","data":{"read_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.956522_279.1","section":"bucket.instance","name":"lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31","timestamp":"2021-02-08T07:24:02.956522086Z","data":{"read_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":1},"write_version":{"tag":"_MMFKePIMeXZBGYRKiNmcRWt","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769042.968947_280.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.968947604Z","data":{"read_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":2},"status":{"status":"write"}}},{"id":"1_1612769042.978250_281.1","section":"bucket.instance","name":"lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32","timestamp":"2021-02-08T07:24:02.978250456Z","data":{"read_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":1},"write_version":{"tag":"_3kbpaQu5ZiMIzD7Ue5aYyt4","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.623875_282.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.623875923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.630835_299.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.630835843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.633069_300.1","section":"bucket","name":"lhddgo-24","timestamp":"2021-02-08T07:24:29.633069001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4t2fafA9W7emjovORdu4vzB","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.637883_301.1","section":"bucket","name":"lhddgo-24","timestamp":"2021-02-08T07:24:29.637883250Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4t2fafA9W7emjovORdu4vzB","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.660183_302.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.660183105Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.665900_303.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.665900158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.667859_304.1","section":"bucket","name":"lhddgo-25","timestamp":"2021-02-08T07:24:29.667859791Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5cBMeEkIR40rErwMHaeqFJ2","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.673799_305.1","section":"bucket","name":"lhddgo-25","timestamp":"2021-02-08T07:24:29.673799031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5cBMeEkIR40rErwMHaeqFJ2","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.720508_306.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.720508838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.726120_307.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.726120869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.727996_308.1","section":"bucket","name":"lhddgo-26","timestamp":"2021-02-08T07:24:29.727996128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__bV0SY4m7IJ85i318hYa99f","ver":1},"status":{"status":"write"}}},{"id":"1_1612769069.732935_309.1","section":"bucket","name":"lhddgo-26","timestamp":"2021-02-08T07:24:29.732935776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__bV0SY4m7IJ85i318hYa99f","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769069.767812_310.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.767812812Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.774076_311.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.774076941Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":1},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.778099_312.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.778099238Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.785267_313.1","section":"bucket.instance","name":"lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33","timestamp":"2021-02-08T07:24:29.785267130Z","data":{"read_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":2},"write_version":{"tag":"_UFZMZVWux_7sLvNLbjsWo-p","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769069.790204_314.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.790204719Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.797562_315.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.797562958Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":1},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.809280_316.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.809280114Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.816992_317.1","section":"bucket.instance","name":"lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34","timestamp":"2021-02-08T07:24:29.816992209Z","data":{"read_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":2},"write_version":{"tag":"_HhjZbERHDIdgEzaiIhmD_By","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769069.828918_318.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.828918200Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"status":{"status":"write"}}},{"id":"1_1612769069.837623_319.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.837623812Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":1},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769069.849480_320.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.849480608Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":3},"status":{"status":"write"}}},{"id":"1_1612769069.857413_321.1","section":"bucket.instance","name":"lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35","timestamp":"2021-02-08T07:24:29.857413152Z","data":{"read_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":2},"write_version":{"tag":"_KTk7gt44pnKn2sidm4l0MI0","ver":3},"status":{"status":"complete"}}},{"id":"1_1612769098.920309_322.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:58.920309067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.927047_339.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:58.927047541Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.929208_340.1","section":"bucket","name":"lhddgo-27","timestamp":"2021-02-08T07:24:58.929208268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E3g7A0Syvz7sJwkowawNoEA","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.934262_341.1","section":"bucket","name":"lhddgo-27","timestamp":"2021-02-08T07:24:58.934262099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E3g7A0Syvz7sJwkowawNoEA","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.961964_342.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:58.961964655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.968667_343.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:58.968667208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769098.970843_344.1","section":"bucket","name":"lhddgo-28","timestamp":"2021-02-08T07:24:58.970843625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yXVDMAYob-zCi8EUwg1b8rK","ver":1},"status":{"status":"write"}}},{"id":"1_1612769098.976100_345.1","section":"bucket","name":"lhddgo-28","timestamp":"2021-02-08T07:24:58.976100609Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yXVDMAYob-zCi8EUwg1b8rK","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.021389_346.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.021389397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.027116_347.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.027116452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.029204_348.1","section":"bucket","name":"lhddgo-29","timestamp":"2021-02-08T07:24:59.029204248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BBdXyLdhOQBxsIZPgiWK1Io","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.040870_349.1","section":"bucket","name":"lhddgo-29","timestamp":"2021-02-08T07:24:59.040870109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BBdXyLdhOQBxsIZPgiWK1Io","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.070082_350.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:59.070082042Z","data":{"read_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.076291_351.1","section":"bucket.instance","name":"lhddgo-27:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36","timestamp":"2021-02-08T07:24:59.076291919Z","data":{"read_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":1},"write_version":{"tag":"_7w2ItBUqXeLnacZ_ZLdao5e","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.128145_352.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:59.128145356Z","data":{"read_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.138654_353.1","section":"bucket.instance","name":"lhddgo-28:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37","timestamp":"2021-02-08T07:24:59.138654984Z","data":{"read_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":1},"write_version":{"tag":"_e831M254-QHtXYXSMeS9xjj","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.195524_354.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.195524900Z","data":{"read_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.205763_355.1","section":"bucket.instance","name":"lhddgo-29:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38","timestamp":"2021-02-08T07:24:59.205763967Z","data":{"read_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":1},"write_version":{"tag":"_k9hJvG787nNhPlhJC_oesaG","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.283799_356.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.283799364Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.289172_357.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.289172513Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.291276_358.1","section":"bucket","name":"lhddgo-30","timestamp":"2021-02-08T07:24:59.291276569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UwjaMGoy_DrUnxaidzTazZd","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.296560_359.1","section":"bucket","name":"lhddgo-30","timestamp":"2021-02-08T07:24:59.296560718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UwjaMGoy_DrUnxaidzTazZd","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.322298_360.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.322298131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.328870_361.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.328870951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.331060_362.1","section":"bucket","name":"lhddgo-31","timestamp":"2021-02-08T07:24:59.331060564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2QZd7jhCZ6WbQQApzYZ6Zuq","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.336978_363.1","section":"bucket","name":"lhddgo-31","timestamp":"2021-02-08T07:24:59.336978349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2QZd7jhCZ6WbQQApzYZ6Zuq","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.384623_364.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.384623961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.390826_365.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.390826118Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.392754_366.1","section":"bucket","name":"lhddgo-32","timestamp":"2021-02-08T07:24:59.392754749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bqm4fq2xDjXcNewXd8Ho5au","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.397952_367.1","section":"bucket","name":"lhddgo-32","timestamp":"2021-02-08T07:24:59.397952631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bqm4fq2xDjXcNewXd8Ho5au","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.425432_368.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.425432089Z","data":{"read_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.431859_369.1","section":"bucket.instance","name":"lhddgo-30:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39","timestamp":"2021-02-08T07:24:59.431859354Z","data":{"read_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":1},"write_version":{"tag":"_774mTKBdigq_0cTaEhFK1Ap","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.481778_370.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.481778111Z","data":{"read_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.492185_371.1","section":"bucket.instance","name":"lhddgo-31:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40","timestamp":"2021-02-08T07:24:59.492185468Z","data":{"read_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":1},"write_version":{"tag":"_aZrqjnVOcbF2RA8h_nnB1aJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.548697_372.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.548697908Z","data":{"read_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.554553_373.1","section":"bucket.instance","name":"lhddgo-32:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41","timestamp":"2021-02-08T07:24:59.554553520Z","data":{"read_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":1},"write_version":{"tag":"_B2YrecULfbkD9GxiwRIVxMu","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769099.625575_374.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.625575517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.632352_375.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.632352418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.634874_376.1","section":"bucket","name":"lhddgo-33","timestamp":"2021-02-08T07:24:59.634874165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S1nyQKgyKti5lpNsSFRpOE5","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.640734_377.1","section":"bucket","name":"lhddgo-33","timestamp":"2021-02-08T07:24:59.640734750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S1nyQKgyKti5lpNsSFRpOE5","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.671312_378.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:24:59.671312553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.679437_379.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:24:59.679437197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.681736_380.1","section":"bucket","name":"lhddgo-34","timestamp":"2021-02-08T07:24:59.681736519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BeJshNgmDKzGicFSITjzLQF","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.686854_381.1","section":"bucket","name":"lhddgo-34","timestamp":"2021-02-08T07:24:59.686854680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BeJshNgmDKzGicFSITjzLQF","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.734811_382.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:24:59.734811285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.740722_383.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:24:59.740722499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.742608_384.1","section":"bucket","name":"lhddgo-35","timestamp":"2021-02-08T07:24:59.742608646Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q-9lHc8CrdYvDkHWoFJqG8X","ver":1},"status":{"status":"write"}}},{"id":"1_1612769099.748237_385.1","section":"bucket","name":"lhddgo-35","timestamp":"2021-02-08T07:24:59.748237598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q-9lHc8CrdYvDkHWoFJqG8X","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769099.776781_386.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.776781310Z","data":{"read_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":2},"status":{"status":"write"}}},{"id":"1_1612769099.783360_387.1","section":"bucket.instance","name":"lhddgo-33:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42","timestamp":"2021-02-08T07:24:59.783360110Z","data":{"read_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":1},"write_version":{"tag":"_r-9oHfM4atTbH0FghuJuz_g","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769100.836953_388.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:25:00.836953168Z","data":{"read_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":2},"status":{"status":"write"}}},{"id":"1_1612769100.848578_389.1","section":"bucket.instance","name":"lhddgo-34:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43","timestamp":"2021-02-08T07:25:00.848578115Z","data":{"read_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":1},"write_version":{"tag":"_NvxWFJWakOScLTZfvjkAF-K","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769102.144127_390.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:25:02.144127048Z","data":{"read_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":2},"status":{"status":"write"}}},{"id":"1_1612769102.156036_391.1","section":"bucket.instance","name":"lhddgo-35:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44","timestamp":"2021-02-08T07:25:02.156036210Z","data":{"read_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":1},"write_version":{"tag":"_zSJLKgROBGZErcJGXTE91Vs","ver":2},"status":{"status":"complete"}}},{"id":"1_1612769102.760731_392.1","section":"bucket.instance","name":"lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45","timestamp":"2021-02-08T07:25:02.760731916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pUmMyR0Isdm2KnONKSzA-Ey","ver":1},"status":{"status":"write"}}},{"id":"1_1612769102.767819_393.1","section":"bucket.instance","name":"lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45","timestamp":"2021-02-08T07:25:02.767819540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pUmMyR0Isdm2KnONKSzA-Ey","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769102.770229_394.1","section":"bucket","name":"lhddgo-36","timestamp":"2021-02-08T07:25:02.770229312Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8bkeVL24JNAUI6gG0schLP","ver":1},"status":{"status":"write"}}},{"id":"1_1612769102.775801_395.1","section":"bucket","name":"lhddgo-36","timestamp":"2021-02-08T07:25:02.775801188Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8bkeVL24JNAUI6gG0schLP","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.117151_396.1","section":"bucket.instance","name":"lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46","timestamp":"2021-02-08T07:25:03.117151764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g5Dq8aX87ZwjcUVPxfbcfaL","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.122644_397.1","section":"bucket.instance","name":"lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46","timestamp":"2021-02-08T07:25:03.122644962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g5Dq8aX87ZwjcUVPxfbcfaL","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.124978_398.1","section":"bucket","name":"lhddgo-37","timestamp":"2021-02-08T07:25:03.124978224Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__2oxvZ6RrkXDCfhasSdxBZQ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.130437_399.1","section":"bucket","name":"lhddgo-37","timestamp":"2021-02-08T07:25:03.130437751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__2oxvZ6RrkXDCfhasSdxBZQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.182592_400.1","section":"bucket.instance","name":"lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47","timestamp":"2021-02-08T07:25:03.182592377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_06zd8oUyALlG1ZsQD8Fdol-","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.187922_401.1","section":"bucket.instance","name":"lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47","timestamp":"2021-02-08T07:25:03.187922879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_06zd8oUyALlG1ZsQD8Fdol-","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769103.189848_402.1","section":"bucket","name":"lhddgo-38","timestamp":"2021-02-08T07:25:03.189848323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUOpEUVHAB91BG5u5pJNnrO","ver":1},"status":{"status":"write"}}},{"id":"1_1612769103.194698_403.1","section":"bucket","name":"lhddgo-38","timestamp":"2021-02-08T07:25:03.194698092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nUOpEUVHAB91BG5u5pJNnrO","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769106.694646_404.1","section":"bucket.instance","name":"lhddgo-39:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48","timestamp":"2021-02-08T07:25:06.694646856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CNAF5lCyovQyVl-qG7T3ANF","ver":1},"status":{"status":"write"}}},{"id":"1_1612769106.699588_406.1","section":"bucket.instance","name":"lhddgo-39:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48","timestamp":"2021-02-08T07:25:06.699588368Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CNAF5lCyovQyVl-qG7T3ANF","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769106.701410_407.1","section":"bucket","name":"lhddgo-39","timestamp":"2021-02-08T07:25:06.701410302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1SwIrqXTWHi7vifEqt55SsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769106.705845_408.1","section":"bucket","name":"lhddgo-39","timestamp":"2021-02-08T07:25:06.705845190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1SwIrqXTWHi7vifEqt55SsQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.036501_409.1","section":"bucket.instance","name":"lhddgo-40:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49","timestamp":"2021-02-08T07:25:07.036501974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dDggL3R9PYxxraDauffrO09","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.042000_410.1","section":"bucket.instance","name":"lhddgo-40:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49","timestamp":"2021-02-08T07:25:07.042000544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dDggL3R9PYxxraDauffrO09","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.043987_411.1","section":"bucket","name":"lhddgo-40","timestamp":"2021-02-08T07:25:07.043987355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZOZ_VlHozV1b8Q-DrynRt61","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.049028_412.1","section":"bucket","name":"lhddgo-40","timestamp":"2021-02-08T07:25:07.049028688Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZOZ_VlHozV1b8Q-DrynRt61","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.097668_413.1","section":"bucket.instance","name":"lhddgo-41:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50","timestamp":"2021-02-08T07:25:07.097668057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTrt6zeu476exuMMzu3hcGZ","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.103981_414.1","section":"bucket.instance","name":"lhddgo-41:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50","timestamp":"2021-02-08T07:25:07.103981325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UTrt6zeu476exuMMzu3hcGZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769107.106208_415.1","section":"bucket","name":"lhddgo-41","timestamp":"2021-02-08T07:25:07.106208500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YtG89TnLF4vYQMJMWzkgy3J","ver":1},"status":{"status":"write"}}},{"id":"1_1612769107.111803_416.1","section":"bucket","name":"lhddgo-41","timestamp":"2021-02-08T07:25:07.111803603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YtG89TnLF4vYQMJMWzkgy3J","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:41.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:41.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:41.923 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 49be1c29-d1bd-4210-9e84-e35ba7fb03cf --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:41.962 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:41.962 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:42.228 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1612769180.247075_295.1","section":"bucket.instance","name":"lhddgo-42:c897876e-5513-44a9-a99f-ee5993968e59.5488.1","timestamp":"2021-02-08T07:26:20.247075123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3sE_OWuzgpOmlS5-fwu2ibC","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.252713_296.1","section":"bucket.instance","name":"lhddgo-42:c897876e-5513-44a9-a99f-ee5993968e59.5488.1","timestamp":"2021-02-08T07:26:20.252713143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3sE_OWuzgpOmlS5-fwu2ibC","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.254430_297.1","section":"bucket","name":"lhddgo-42","timestamp":"2021-02-08T07:26:20.254430663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JoD1S67WuuGgOVJpvhkR8CB","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.259058_298.1","section":"bucket","name":"lhddgo-42","timestamp":"2021-02-08T07:26:20.259058785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JoD1S67WuuGgOVJpvhkR8CB","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.304855_299.1","section":"bucket.instance","name":"lhddgo-43:c897876e-5513-44a9-a99f-ee5993968e59.5488.2","timestamp":"2021-02-08T07:26:20.304855834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PIRGM9f_G1__bOqQrq2IG8M","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.310605_300.1","section":"bucket.instance","name":"lhddgo-43:c897876e-5513-44a9-a99f-ee5993968e59.5488.2","timestamp":"2021-02-08T07:26:20.310605892Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PIRGM9f_G1__bOqQrq2IG8M","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.312362_301.1","section":"bucket","name":"lhddgo-43","timestamp":"2021-02-08T07:26:20.312362299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rswTOwdBUTXoF2hxRXKgNL9","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.317275_302.1","section":"bucket","name":"lhddgo-43","timestamp":"2021-02-08T07:26:20.317275065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rswTOwdBUTXoF2hxRXKgNL9","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:42.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:42.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:42.229 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 49be1c29-d1bd-4210-9e84-e35ba7fb03cf --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:42.267 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:42.267 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:42.506 INFO:teuthology.orchestra.run.smithi110.stdout:[{"id":"1_1612769180.247075_295.1","section":"bucket.instance","name":"lhddgo-42:c897876e-5513-44a9-a99f-ee5993968e59.5488.1","timestamp":"2021-02-08T07:26:20.247075123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3sE_OWuzgpOmlS5-fwu2ibC","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.252713_296.1","section":"bucket.instance","name":"lhddgo-42:c897876e-5513-44a9-a99f-ee5993968e59.5488.1","timestamp":"2021-02-08T07:26:20.252713143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3sE_OWuzgpOmlS5-fwu2ibC","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.254430_297.1","section":"bucket","name":"lhddgo-42","timestamp":"2021-02-08T07:26:20.254430663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JoD1S67WuuGgOVJpvhkR8CB","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.259058_298.1","section":"bucket","name":"lhddgo-42","timestamp":"2021-02-08T07:26:20.259058785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JoD1S67WuuGgOVJpvhkR8CB","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.304855_299.1","section":"bucket.instance","name":"lhddgo-43:c897876e-5513-44a9-a99f-ee5993968e59.5488.2","timestamp":"2021-02-08T07:26:20.304855834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PIRGM9f_G1__bOqQrq2IG8M","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.310605_300.1","section":"bucket.instance","name":"lhddgo-43:c897876e-5513-44a9-a99f-ee5993968e59.5488.2","timestamp":"2021-02-08T07:26:20.310605892Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PIRGM9f_G1__bOqQrq2IG8M","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.312362_301.1","section":"bucket","name":"lhddgo-43","timestamp":"2021-02-08T07:26:20.312362299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rswTOwdBUTXoF2hxRXKgNL9","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.317275_302.1","section":"bucket","name":"lhddgo-43","timestamp":"2021-02-08T07:26:20.317275065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rswTOwdBUTXoF2hxRXKgNL9","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:42.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:42.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:42.507 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 49be1c29-d1bd-4210-9e84-e35ba7fb03cf --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:42.546 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:42.546 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:42.816 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1612769180.247075_295.1","section":"bucket.instance","name":"lhddgo-42:c897876e-5513-44a9-a99f-ee5993968e59.5488.1","timestamp":"2021-02-08T07:26:20.247075123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3sE_OWuzgpOmlS5-fwu2ibC","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.252713_296.1","section":"bucket.instance","name":"lhddgo-42:c897876e-5513-44a9-a99f-ee5993968e59.5488.1","timestamp":"2021-02-08T07:26:20.252713143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3sE_OWuzgpOmlS5-fwu2ibC","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.254430_297.1","section":"bucket","name":"lhddgo-42","timestamp":"2021-02-08T07:26:20.254430663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JoD1S67WuuGgOVJpvhkR8CB","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.259058_298.1","section":"bucket","name":"lhddgo-42","timestamp":"2021-02-08T07:26:20.259058785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JoD1S67WuuGgOVJpvhkR8CB","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.304855_299.1","section":"bucket.instance","name":"lhddgo-43:c897876e-5513-44a9-a99f-ee5993968e59.5488.2","timestamp":"2021-02-08T07:26:20.304855834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PIRGM9f_G1__bOqQrq2IG8M","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.310605_300.1","section":"bucket.instance","name":"lhddgo-43:c897876e-5513-44a9-a99f-ee5993968e59.5488.2","timestamp":"2021-02-08T07:26:20.310605892Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PIRGM9f_G1__bOqQrq2IG8M","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.312362_301.1","section":"bucket","name":"lhddgo-43","timestamp":"2021-02-08T07:26:20.312362299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rswTOwdBUTXoF2hxRXKgNL9","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.317275_302.1","section":"bucket","name":"lhddgo-43","timestamp":"2021-02-08T07:26:20.317275065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rswTOwdBUTXoF2hxRXKgNL9","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:42.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:42.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:42.817 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 49be1c29-d1bd-4210-9e84-e35ba7fb03cf --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:43.088 INFO:teuthology.orchestra.run.smithi110.stdout:[{"id":"1_1612769180.247075_295.1","section":"bucket.instance","name":"lhddgo-42:c897876e-5513-44a9-a99f-ee5993968e59.5488.1","timestamp":"2021-02-08T07:26:20.247075123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3sE_OWuzgpOmlS5-fwu2ibC","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.252713_296.1","section":"bucket.instance","name":"lhddgo-42:c897876e-5513-44a9-a99f-ee5993968e59.5488.1","timestamp":"2021-02-08T07:26:20.252713143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3sE_OWuzgpOmlS5-fwu2ibC","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.254430_297.1","section":"bucket","name":"lhddgo-42","timestamp":"2021-02-08T07:26:20.254430663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JoD1S67WuuGgOVJpvhkR8CB","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.259058_298.1","section":"bucket","name":"lhddgo-42","timestamp":"2021-02-08T07:26:20.259058785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JoD1S67WuuGgOVJpvhkR8CB","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.304855_299.1","section":"bucket.instance","name":"lhddgo-43:c897876e-5513-44a9-a99f-ee5993968e59.5488.2","timestamp":"2021-02-08T07:26:20.304855834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PIRGM9f_G1__bOqQrq2IG8M","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.310605_300.1","section":"bucket.instance","name":"lhddgo-43:c897876e-5513-44a9-a99f-ee5993968e59.5488.2","timestamp":"2021-02-08T07:26:20.310605892Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PIRGM9f_G1__bOqQrq2IG8M","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769180.312362_301.1","section":"bucket","name":"lhddgo-43","timestamp":"2021-02-08T07:26:20.312362299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rswTOwdBUTXoF2hxRXKgNL9","ver":1},"status":{"status":"write"}}},{"id":"1_1612769180.317275_302.1","section":"bucket","name":"lhddgo-43","timestamp":"2021-02-08T07:26:20.317275065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rswTOwdBUTXoF2hxRXKgNL9","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:43.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:43.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:43.089 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 38375dfb-4fb1-40a7-81e1-7d3f26b8d208 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:43.130 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:43.130 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:43.402 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1612769246.742325_232.1","section":"bucket.instance","name":"lhddgo-44:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1","timestamp":"2021-02-08T07:27:26.742325186Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XD4UYmwXng2kARbyIq7QHL5","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.747469_233.1","section":"bucket.instance","name":"lhddgo-44:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1","timestamp":"2021-02-08T07:27:26.747469809Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XD4UYmwXng2kARbyIq7QHL5","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.749562_234.1","section":"bucket","name":"lhddgo-44","timestamp":"2021-02-08T07:27:26.749562146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pV4oEqybOSXlXxC-s9bs9rM","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.755298_235.1","section":"bucket","name":"lhddgo-44","timestamp":"2021-02-08T07:27:26.755298017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pV4oEqybOSXlXxC-s9bs9rM","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.798077_236.1","section":"bucket.instance","name":"lhddgo-45:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2","timestamp":"2021-02-08T07:27:26.798077905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l5OpuGbNrzDnc1SjSRoaBWt","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.803924_237.1","section":"bucket.instance","name":"lhddgo-45:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2","timestamp":"2021-02-08T07:27:26.803924258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l5OpuGbNrzDnc1SjSRoaBWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.805885_238.1","section":"bucket","name":"lhddgo-45","timestamp":"2021-02-08T07:27:26.805885908Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_guvL_56y3jR_zi_5u90-QeN","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.810829_239.1","section":"bucket","name":"lhddgo-45","timestamp":"2021-02-08T07:27:26.810829258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_guvL_56y3jR_zi_5u90-QeN","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:43.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:43.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:43.403 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 38375dfb-4fb1-40a7-81e1-7d3f26b8d208 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:43.441 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:43.441 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:43.660 INFO:teuthology.orchestra.run.smithi110.stdout:[{"id":"1_1612769246.742325_232.1","section":"bucket.instance","name":"lhddgo-44:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1","timestamp":"2021-02-08T07:27:26.742325186Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XD4UYmwXng2kARbyIq7QHL5","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.747469_233.1","section":"bucket.instance","name":"lhddgo-44:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1","timestamp":"2021-02-08T07:27:26.747469809Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XD4UYmwXng2kARbyIq7QHL5","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.749562_234.1","section":"bucket","name":"lhddgo-44","timestamp":"2021-02-08T07:27:26.749562146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pV4oEqybOSXlXxC-s9bs9rM","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.755298_235.1","section":"bucket","name":"lhddgo-44","timestamp":"2021-02-08T07:27:26.755298017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pV4oEqybOSXlXxC-s9bs9rM","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.798077_236.1","section":"bucket.instance","name":"lhddgo-45:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2","timestamp":"2021-02-08T07:27:26.798077905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l5OpuGbNrzDnc1SjSRoaBWt","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.803924_237.1","section":"bucket.instance","name":"lhddgo-45:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2","timestamp":"2021-02-08T07:27:26.803924258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l5OpuGbNrzDnc1SjSRoaBWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.805885_238.1","section":"bucket","name":"lhddgo-45","timestamp":"2021-02-08T07:27:26.805885908Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_guvL_56y3jR_zi_5u90-QeN","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.810829_239.1","section":"bucket","name":"lhddgo-45","timestamp":"2021-02-08T07:27:26.810829258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_guvL_56y3jR_zi_5u90-QeN","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:43.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:43.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:43.661 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 38375dfb-4fb1-40a7-81e1-7d3f26b8d208 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:43.700 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:43.700 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:43.971 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1612769246.742325_232.1","section":"bucket.instance","name":"lhddgo-44:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1","timestamp":"2021-02-08T07:27:26.742325186Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XD4UYmwXng2kARbyIq7QHL5","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.747469_233.1","section":"bucket.instance","name":"lhddgo-44:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1","timestamp":"2021-02-08T07:27:26.747469809Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XD4UYmwXng2kARbyIq7QHL5","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.749562_234.1","section":"bucket","name":"lhddgo-44","timestamp":"2021-02-08T07:27:26.749562146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pV4oEqybOSXlXxC-s9bs9rM","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.755298_235.1","section":"bucket","name":"lhddgo-44","timestamp":"2021-02-08T07:27:26.755298017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pV4oEqybOSXlXxC-s9bs9rM","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.798077_236.1","section":"bucket.instance","name":"lhddgo-45:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2","timestamp":"2021-02-08T07:27:26.798077905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l5OpuGbNrzDnc1SjSRoaBWt","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.803924_237.1","section":"bucket.instance","name":"lhddgo-45:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2","timestamp":"2021-02-08T07:27:26.803924258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l5OpuGbNrzDnc1SjSRoaBWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.805885_238.1","section":"bucket","name":"lhddgo-45","timestamp":"2021-02-08T07:27:26.805885908Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_guvL_56y3jR_zi_5u90-QeN","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.810829_239.1","section":"bucket","name":"lhddgo-45","timestamp":"2021-02-08T07:27:26.810829258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_guvL_56y3jR_zi_5u90-QeN","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:43.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:43.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:43.972 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 38375dfb-4fb1-40a7-81e1-7d3f26b8d208 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:44.251 INFO:teuthology.orchestra.run.smithi110.stdout:[{"id":"1_1612769246.742325_232.1","section":"bucket.instance","name":"lhddgo-44:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1","timestamp":"2021-02-08T07:27:26.742325186Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XD4UYmwXng2kARbyIq7QHL5","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.747469_233.1","section":"bucket.instance","name":"lhddgo-44:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1","timestamp":"2021-02-08T07:27:26.747469809Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XD4UYmwXng2kARbyIq7QHL5","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.749562_234.1","section":"bucket","name":"lhddgo-44","timestamp":"2021-02-08T07:27:26.749562146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pV4oEqybOSXlXxC-s9bs9rM","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.755298_235.1","section":"bucket","name":"lhddgo-44","timestamp":"2021-02-08T07:27:26.755298017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pV4oEqybOSXlXxC-s9bs9rM","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.798077_236.1","section":"bucket.instance","name":"lhddgo-45:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2","timestamp":"2021-02-08T07:27:26.798077905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l5OpuGbNrzDnc1SjSRoaBWt","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.803924_237.1","section":"bucket.instance","name":"lhddgo-45:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2","timestamp":"2021-02-08T07:27:26.803924258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l5OpuGbNrzDnc1SjSRoaBWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1612769246.805885_238.1","section":"bucket","name":"lhddgo-45","timestamp":"2021-02-08T07:27:26.805885908Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_guvL_56y3jR_zi_5u90-QeN","ver":1},"status":{"status":"write"}}},{"id":"1_1612769246.810829_239.1","section":"bucket","name":"lhddgo-45","timestamp":"2021-02-08T07:27:26.810829258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_guvL_56y3jR_zi_5u90-QeN","ver":1},"status":{"status":"complete"}}}] 2021-02-08T07:27:44.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:27:44.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:27:44.252 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:27:44.290 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:44.291 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:44.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:27:44.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:27:44.611 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:27:44.641 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:44.642 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:45.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:27:45.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:27:45.019 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:27:45.058 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:45.059 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:45.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:27:45.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:27:45.354 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:27:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:45.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:45.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:45.724 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period d4b11cd5-1a7c-4383-afcf-402844a1791d --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:45.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:45.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:46.036 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:27:46.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:46.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:46.037 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period d4b11cd5-1a7c-4383-afcf-402844a1791d --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:46.073 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:46.074 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:46.320 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2021-02-08T07:27:46.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:46.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:46.320 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period d4b11cd5-1a7c-4383-afcf-402844a1791d --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:46.360 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:46.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:46.621 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:27:46.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:46.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'd4b11cd5-1a7c-4383-afcf-402844a1791d', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:46.622 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period d4b11cd5-1a7c-4383-afcf-402844a1791d --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:46.656 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:46.656 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:46.887 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2021-02-08T07:27:46.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:46.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:46.888 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 49be1c29-d1bd-4210-9e84-e35ba7fb03cf --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:46.925 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:46.926 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:47.183 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:27:47.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:47.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:47.184 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 49be1c29-d1bd-4210-9e84-e35ba7fb03cf --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:47.221 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:47.221 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:47.457 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2021-02-08T07:27:47.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:47.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:47.458 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 49be1c29-d1bd-4210-9e84-e35ba7fb03cf --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:47.498 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:47.784 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:27:47.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:47.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '49be1c29-d1bd-4210-9e84-e35ba7fb03cf', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:47.785 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 49be1c29-d1bd-4210-9e84-e35ba7fb03cf --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:47.820 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:48.057 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2021-02-08T07:27:48.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:48.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:48.058 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 38375dfb-4fb1-40a7-81e1-7d3f26b8d208 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:48.097 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:48.097 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:48.368 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:27:48.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:48.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:48.369 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 38375dfb-4fb1-40a7-81e1-7d3f26b8d208 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:48.406 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:48.406 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:48.647 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2021-02-08T07:27:48.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:48.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:48.648 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 38375dfb-4fb1-40a7-81e1-7d3f26b8d208 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:48.685 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:48.685 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:48.961 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:27:48.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:48.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '38375dfb-4fb1-40a7-81e1-7d3f26b8d208', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:48.962 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 38375dfb-4fb1-40a7-81e1-7d3f26b8d208 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:49.000 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:49.001 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:49.238 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2021-02-08T07:27:49.239 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2021-02-08T07:27:49.239 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-46 2021-02-08T07:27:50.103 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-47 2021-02-08T07:27:50.301 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-48 2021-02-08T07:27:50.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:50.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:50.531 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:50.570 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:50.571 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:50.825 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769107.395761_173.1","last_update":"2021-02-08T07:25:07.395761Z"},{"marker":"1_1612768880.006161_251.3","last_update":"2021-02-08T07:21:20.006161Z"},{"marker":"1_1612769270.389039_398.1","last_update":"2021-02-08T07:27:50.389039Z"},{"marker":"1_1612768924.012632_160.1","last_update":"2021-02-08T07:22:04.012632Z"}] 2021-02-08T07:27:50.825 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-08T07:27:50.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:50.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:50.826 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:50.870 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:50.870 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:51.114 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769100.038150_371.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:00.038150Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.012632_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.012632Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:27:51.114 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=2 master=1_1612769270.389039_398.1 target=1_1612769100.038150_371.2 2021-02-08T07:27:56.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:56.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:56.116 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:56.157 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:56.158 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:56.401 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.012632_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.012632Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:27:56.401 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-08T07:27:56.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:56.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:56.402 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:56.442 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:56.443 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:56.710 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769107.395761_173.1","last_update":"2021-02-08T07:25:07.395761Z"},{"marker":"1_1612768880.006161_251.3","last_update":"2021-02-08T07:21:20.006161Z"},{"marker":"1_1612769271.789494_416.1","last_update":"2021-02-08T07:27:51.789494Z"},{"marker":"1_1612768924.012632_160.1","last_update":"2021-02-08T07:22:04.012632Z"}] 2021-02-08T07:27:56.711 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-08T07:27:56.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:56.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:56.712 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:56.796 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:56.796 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:57.052 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7430472880135713257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.012632_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.012632Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:27:57.052 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-08T07:27:57.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:57.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:57.053 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:57.132 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:57.132 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:57.389 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769107.395761_173.1","last_update":"2021-02-08T07:25:07.395761Z"},{"marker":"1_1612768880.006161_251.3","last_update":"2021-02-08T07:21:20.006161Z"},{"marker":"1_1612769271.789494_416.1","last_update":"2021-02-08T07:27:51.789494Z"},{"marker":"1_1612768924.012632_160.1","last_update":"2021-02-08T07:22:04.012632Z"}] 2021-02-08T07:27:57.389 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-08T07:27:57.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:57.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:57.390 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:57.431 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:57.431 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:57.707 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":729464676215090460},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.012632_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.012632Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:27:57.707 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-08T07:27:57.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:57.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:57.708 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:57.789 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:57.789 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:58.027 INFO:teuthology.orchestra.run.smithi110.stdout:[{"marker":"1_1612769107.392980_109.1","last_update":"2021-02-08T07:25:07.392980Z"},{"marker":"1_1612768881.007212_292.3","last_update":"2021-02-08T07:21:21.007212Z"},{"marker":"1_1612769271.739830_424.1","last_update":"2021-02-08T07:27:51.739830Z"},{"marker":"1_1612768925.016683_163.1","last_update":"2021-02-08T07:22:05.016683Z"}] 2021-02-08T07:27:58.027 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-08T07:27:58.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:58.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:58.027 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:58.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:58.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:58.339 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768925.016683_163.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:05.016683Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:27:58.339 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-08T07:27:58.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:58.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:58.340 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:58.380 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:58.381 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:58.620 INFO:teuthology.orchestra.run.smithi110.stdout:[{"marker":"1_1612769107.392980_109.1","last_update":"2021-02-08T07:25:07.392980Z"},{"marker":"1_1612768881.007212_292.3","last_update":"2021-02-08T07:21:21.007212Z"},{"marker":"1_1612769271.739830_424.1","last_update":"2021-02-08T07:27:51.739830Z"},{"marker":"1_1612768925.016683_163.1","last_update":"2021-02-08T07:22:05.016683Z"}] 2021-02-08T07:27:58.620 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-08T07:27:58.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:58.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:58.621 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:58.664 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:58.664 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:58.923 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":711522086589135958},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768925.016683_163.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:05.016683Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:27:58.923 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-08T07:27:58.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:58.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:58.924 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:59.002 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:59.003 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:59.254 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769106.878997_105.1","last_update":"2021-02-08T07:25:06.878997Z"},{"marker":"1_1612768880.010730_219.3","last_update":"2021-02-08T07:21:20.010730Z"},{"marker":"1_1612769271.788832_344.1","last_update":"2021-02-08T07:27:51.788832Z"},{"marker":"1_1612768924.017986_97.1","last_update":"2021-02-08T07:22:04.017986Z"}] 2021-02-08T07:27:59.254 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-08T07:27:59.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:59.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:59.255 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:59.337 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:59.337 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:59.612 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12652603110286941624},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769106.878997_105.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:06.878997Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.010730_219.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.010730Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.788832_344.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.788832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.017986_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.017986Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:27:59.612 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-08T07:27:59.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:59.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:59.613 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:27:59.692 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:27:59.692 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:27:59.958 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769106.878997_105.1","last_update":"2021-02-08T07:25:06.878997Z"},{"marker":"1_1612768880.010730_219.3","last_update":"2021-02-08T07:21:20.010730Z"},{"marker":"1_1612769271.788832_344.1","last_update":"2021-02-08T07:27:51.788832Z"},{"marker":"1_1612768924.017986_97.1","last_update":"2021-02-08T07:22:04.017986Z"}] 2021-02-08T07:27:59.958 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-08T07:27:59.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:59.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:27:59.959 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:28:00.002 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:28:00.002 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:28:00.237 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5378454525578809946},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769106.878997_105.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:06.878997Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.010730_219.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.010730Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.788832_344.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.788832Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612768924.017986_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:22:04.017986Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:28:00.238 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-08T07:28:00.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:00.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:00.238 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:28:00.273 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:28:00.273 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:28:00.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:28:00.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:28:00.542 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:28:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:28:00.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:28:00.869 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:28:00.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:00.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:00.870 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:28:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:28:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:28:01.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:28:01.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:28:01.136 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:28:01.214 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:28:01.215 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:28:01.449 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2021-02-08T07:28:01.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:01.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:01.450 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:28:01.486 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:28:01.486 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:28:01.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:28:01.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:28:01.762 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:28:01.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:28:01.855 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:28:02.127 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2021-02-08T07:28:02.128 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2021-02-08T07:28:02.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:02.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:02.129 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:28:02.168 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:28:02.168 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:28:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:28:02.189+0000 7f8b64018080 0 failed reading obj info from .rgw.root:zone_info.757db88c-ee56-405d-bd0c-79b9f5b05722: (2) No such file or directory 2021-02-08T07:28:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:28:02.189+0000 7f8b64018080 0 WARNING: could not read zone params for zone id=757db88c-ee56-405d-bd0c-79b9f5b05722 name=test-zone1 2021-02-08T07:28:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:28:02.189+0000 7f8b64018080 0 failed reading obj info from .rgw.root:zone_info.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d: (2) No such file or directory 2021-02-08T07:28:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:28:02.189+0000 7f8b64018080 0 WARNING: could not read zone params for zone id=8e9c5146-39d0-40f8-b41d-7c27e66f6e3d name=test-zone3 2021-02-08T07:28:02.221 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:28:02.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:02.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:28:02.222 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:28:02.306 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:28:02.306 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:28:02.558 INFO:teuthology.orchestra.run.smithi110.stderr:Sending period to new master zone 757db88c-ee56-405d-bd0c-79b9f5b05722 2021-02-08T07:28:03.649 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","epoch":2,"predecessor_uuid":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","sync_status":["","","","1_1612769180.317275_302.1"],"period_map":{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":4} 2021-02-08T07:28:03.650 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-02-08T07:29:03.650 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-49 2021-02-08T07:29:03.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:03.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:03.751 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:03.789 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:03.790 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:04.018 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=38375dfb-4fb1-40a7-81e1-7d3f26b8d208 2021-02-08T07:29:04.053 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612769343.699091_290.1","last_update":"2021-02-08T07:29:03.699091Z"}] 2021-02-08T07:29:04.054 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:29:04.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:04.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:04.054 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:04.095 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:04.095 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:04.328 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769270.349899_268.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:50.349899Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:29:04.328 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769343.699091_290.1 target=1_1612769270.349899_268.1 2021-02-08T07:29:09.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:09.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:09.330 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:09.603 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769270.349899_268.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:50.349899Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:29:09.603 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769343.699091_290.1 target=1_1612769270.349899_268.1 2021-02-08T07:29:14.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:14.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:14.605 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:14.645 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:14.645 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:14.879 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769270.349899_268.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:50.349899Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:29:14.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769343.699091_290.1 target=1_1612769270.349899_268.1 2021-02-08T07:29:19.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:19.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:19.881 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:19.923 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:19.923 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:20.157 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769270.349899_268.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:50.349899Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:29:20.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769343.699091_290.1 target=1_1612769270.349899_268.1 2021-02-08T07:29:25.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:25.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:25.159 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:25.201 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:25.437 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769270.349899_268.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:50.349899Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:29:25.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769343.699091_290.1 target=1_1612769270.349899_268.1 2021-02-08T07:29:30.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:30.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:30.439 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:30.481 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:30.481 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:30.714 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769270.349899_268.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:50.349899Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:29:30.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1612769343.699091_290.1 target=1_1612769270.349899_268.1 2021-02-08T07:29:34.872 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:29:34.863+0000 7f763e7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:29:34.874 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:29:34.870+0000 7f72d2ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:29:35.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:35.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:35.715 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:35.759 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:35.759 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:35.984 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769343.699091_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:29:03.699091Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:29:35.984 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:29:35.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:29:35.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:35.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:35.985 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:36.036 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:36.036 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:36.301 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769343.699091_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:29:03.699091Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:29:36.301 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:29:36.302 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:29:36.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:36.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:29:36.302 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:29:36.343 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:36.343 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:36.619 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769343.699091_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:29:03.699091Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:29:36.620 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:29:36.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=757db88c-ee56-405d-bd0c-79b9f5b05722', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:29:36.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=757db88c-ee56-405d-bd0c-79b9f5b05722', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:29:36.632 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=757db88c-ee56-405d-bd0c-79b9f5b05722 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:29:36.700 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:36.700 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:36.729 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:29:36.719+0000 7f090ff18080 0 failed reading obj info from .rgw.root:zone_info.757db88c-ee56-405d-bd0c-79b9f5b05722: (2) No such file or directory 2021-02-08T07:29:36.729 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:29:36.719+0000 7f090ff18080 0 WARNING: could not read zone params for zone id=757db88c-ee56-405d-bd0c-79b9f5b05722 name=test-zone1 2021-02-08T07:29:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:29:36.719+0000 7f090ff18080 0 failed reading obj info from .rgw.root:zone_info.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d: (2) No such file or directory 2021-02-08T07:29:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:29:36.719+0000 7f090ff18080 0 WARNING: could not read zone params for zone id=8e9c5146-39d0-40f8-b41d-7c27e66f6e3d name=test-zone3 2021-02-08T07:29:36.742 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:29:36.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:29:36.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:29:36.743 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:29:36.826 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:29:36.826 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:29:37.056 INFO:teuthology.orchestra.run.smithi110.stderr:Sending period to new master zone 757db88c-ee56-405d-bd0c-79b9f5b05722 2021-02-08T07:29:38.165 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","epoch":3,"predecessor_uuid":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","sync_status":["","","","1_1612769180.317275_302.1"],"period_map":{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":4} 2021-02-08T07:29:38.165 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "757db88c-ee56-405d-bd0c-79b9f5b05722" 2021-02-08T07:30:38.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:30:38.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:30:38.508 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:30:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:30:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:30:38.577 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:30:38.565+0000 7f5d43ccf080 0 failed reading obj info from .rgw.root:zone_info.757db88c-ee56-405d-bd0c-79b9f5b05722: (2) No such file or directory 2021-02-08T07:30:38.577 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:30:38.565+0000 7f5d43ccf080 0 WARNING: could not read zone params for zone id=757db88c-ee56-405d-bd0c-79b9f5b05722 name=test-zone1 2021-02-08T07:30:38.577 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:30:38.565+0000 7f5d43ccf080 0 failed reading obj info from .rgw.root:zone_info.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d: (2) No such file or directory 2021-02-08T07:30:38.577 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:30:38.565+0000 7f5d43ccf080 0 WARNING: could not read zone params for zone id=8e9c5146-39d0-40f8-b41d-7c27e66f6e3d name=test-zone3 2021-02-08T07:30:38.589 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:30:38.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:30:38.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:30:38.590 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:30:38.669 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:30:38.669 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:30:38.902 INFO:teuthology.orchestra.run.smithi110.stderr:Sending period to new master zone 757db88c-ee56-405d-bd0c-79b9f5b05722 2021-02-08T07:30:40.011 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","epoch":4,"predecessor_uuid":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","sync_status":["","","","1_1612769180.317275_302.1"],"period_map":{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":4} 2021-02-08T07:30:40.012 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2021-02-08T07:31:40.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:31:40.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:31:40.014 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:31:40.056 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:31:40.057 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:31:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:31:40.072+0000 7f51f44b5080 0 failed reading obj info from .rgw.root:zone_info.757db88c-ee56-405d-bd0c-79b9f5b05722: (2) No such file or directory 2021-02-08T07:31:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:31:40.072+0000 7f51f44b5080 0 WARNING: could not read zone params for zone id=757db88c-ee56-405d-bd0c-79b9f5b05722 name=test-zone1 2021-02-08T07:31:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:31:40.072+0000 7f51f44b5080 0 failed reading obj info from .rgw.root:zone_info.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d: (2) No such file or directory 2021-02-08T07:31:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:31:40.072+0000 7f51f44b5080 0 WARNING: could not read zone params for zone id=8e9c5146-39d0-40f8-b41d-7c27e66f6e3d name=test-zone3 2021-02-08T07:31:40.098 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:31:40.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:31:40.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:31:40.099 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:31:40.177 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:31:40.177 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:31:40.428 INFO:teuthology.orchestra.run.smithi110.stderr:Sending period to new master zone 757db88c-ee56-405d-bd0c-79b9f5b05722 2021-02-08T07:31:41.525 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","epoch":5,"predecessor_uuid":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","sync_status":["","","","1_1612769180.317275_302.1"],"period_map":{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":4} 2021-02-08T07:31:41.526 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2021-02-08T07:32:41.527 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2021-02-08T07:32:41.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:32:41.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:32:41.529 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:32:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:41.565 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:41.768 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=38375dfb-4fb1-40a7-81e1-7d3f26b8d208 2021-02-08T07:32:41.807 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612769343.699091_290.1","last_update":"2021-02-08T07:29:03.699091Z"}] 2021-02-08T07:32:41.808 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:32:41.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:32:41.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:32:41.808 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:32:41.848 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:41.848 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:42.091 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769343.699091_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:29:03.699091Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:32:42.092 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:32:42.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:32:42.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:32:42.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:32:42.092 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:32:42.134 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:42.134 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:42.397 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769343.699091_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:29:03.699091Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:32:42.397 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:32:42.397 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:32:42.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:32:42.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:32:42.398 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:32:42.440 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:42.440 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:42.718 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769343.699091_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:29:03.699091Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:32:42.719 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:32:42.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:42.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'SOBS5D2YHN3GGT4Z', '--secret', '1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:42.719 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key SOBS5D2YHN3GGT4Z --secret 1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:42.802 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:42.803 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:42.869 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:32:42.858+0000 7fd19bc01080 0 failed reading obj info from .rgw.root:zone_info.757db88c-ee56-405d-bd0c-79b9f5b05722: (2) No such file or directory 2021-02-08T07:32:42.869 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:32:42.858+0000 7fd19bc01080 0 WARNING: could not read zone params for zone id=757db88c-ee56-405d-bd0c-79b9f5b05722 name=test-zone1 2021-02-08T07:32:42.869 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:32:42.858+0000 7fd19bc01080 0 failed reading obj info from .rgw.root:zone_info.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d: (2) No such file or directory 2021-02-08T07:32:42.869 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:32:42.858+0000 7fd19bc01080 0 WARNING: could not read zone params for zone id=8e9c5146-39d0-40f8-b41d-7c27e66f6e3d name=test-zone3 2021-02-08T07:32:42.881 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"0bd1ebe2-f5f7-4225-9385-7d5a8d9e521b","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"SOBS5D2YHN3GGT4Z","secret_key":"1dNgMZXwSXpLUwSeAQcL1Sx8ixFOQQHG"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:32:42.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:42.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:42.882 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:50.656 INFO:teuthology.orchestra.run.smithi110.stderr:Sending period to new master zone 757db88c-ee56-405d-bd0c-79b9f5b05722 2021-02-08T07:32:51.757 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","epoch":6,"predecessor_uuid":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","sync_status":["","","","1_1612769180.317275_302.1"],"period_map":{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"0bd1ebe2-f5f7-4225-9385-7d5a8d9e521b","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0bd1ebe2-f5f7-4225-9385-7d5a8d9e521b","val":2712861259},{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":4} 2021-02-08T07:32:51.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:51.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:51.758 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:51.799 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:51.799 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:51.821 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:32:51.816+0000 7fdaaab39080 0 failed reading obj info from .rgw.root:zone_info.4c93366d-9954-4fa2-95e3-91873db27ab5: (2) No such file or directory 2021-02-08T07:32:51.822 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:32:51.816+0000 7fdaaab39080 0 WARNING: could not read zone params for zone id=4c93366d-9954-4fa2-95e3-91873db27ab5 name=test-zone4 2021-02-08T07:32:51.824 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:32:51.816+0000 7fdaaab39080 0 failed reading obj info from .rgw.root:zone_info.c897876e-5513-44a9-a99f-ee5993968e59: (2) No such file or directory 2021-02-08T07:32:51.824 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:32:51.816+0000 7fdaaab39080 0 WARNING: could not read zone params for zone id=c897876e-5513-44a9-a99f-ee5993968e59 name=test-zone2 2021-02-08T07:32:51.832 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}} 2021-02-08T07:32:51.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:51.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:51.833 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:51.921 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:51.921 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:51.942 INFO:teuthology.orchestra.run.smithi017.stderr:zone name remove not found in zonegroup test-zonegroup 2021-02-08T07:32:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-08T07:32:51.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:51.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:51.949 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:51.985 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:51.986 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:32:52.034+0000 7f925032f080 0 failed reading obj info from .rgw.root:zone_info.757db88c-ee56-405d-bd0c-79b9f5b05722: (2) No such file or directory 2021-02-08T07:32:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:32:52.034+0000 7f925032f080 0 WARNING: could not read zone params for zone id=757db88c-ee56-405d-bd0c-79b9f5b05722 name=test-zone1 2021-02-08T07:32:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:32:52.034+0000 7f925032f080 0 failed reading obj info from .rgw.root:zone_info.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d: (2) No such file or directory 2021-02-08T07:32:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:32:52.034+0000 7f925032f080 0 WARNING: could not read zone params for zone id=8e9c5146-39d0-40f8-b41d-7c27e66f6e3d name=test-zone3 2021-02-08T07:32:52.051 INFO:teuthology.orchestra.run.smithi110.stderr:2021-02-08T07:32:52.038+0000 7f925032f080 0 zone id 0bd1ebe2-f5f7-4225-9385-7d5a8d9e521b is not a part of zonegroup default 2021-02-08T07:32:52.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:52.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:52.066 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:52.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:52.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:52.180 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:32:52.172+0000 7fc4e2f7f080 0 WARNING: can't generate connection for zone 0bd1ebe2-f5f7-4225-9385-7d5a8d9e521b id remove: no endpoints defined 2021-02-08T07:32:52.397 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:32:52.386+0000 7f7541ffb700 -1 failed to decode period 2021-02-08T07:32:52.397 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:32:52.386+0000 7f85a0ff9700 -1 failed to decode period 2021-02-08T07:32:52.398 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:32:52.392+0000 7f71d67fc700 -1 failed to decode period 2021-02-08T07:32:52.427 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","epoch":7,"predecessor_uuid":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","sync_status":["","","","1_1612769180.317275_302.1"],"period_map":{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":4} 2021-02-08T07:32:52.427 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2021-02-08T07:32:52.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:52.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:52.428 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:52.504 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:52.504 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:52.539 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"08dea35b-2403-4c18-b756-e3c67ec9c843","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}} 2021-02-08T07:32:52.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:52.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:52.540 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:52.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:52.622 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:52.701 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"5693045d-2931-4a45-9a5b-89f6a5a78652","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7"} 2021-02-08T07:32:52.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:52.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:52.702 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:52.786 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:52.786 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:53.133 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","epoch":8,"predecessor_uuid":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","sync_status":["","","","1_1612769180.317275_302.1"],"period_map":{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","zonegroups":[{"id":"08dea35b-2403-4c18-b756-e3c67ec9c843","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"5693045d-2931-4a45-9a5b-89f6a5a78652","zones":[{"id":"5693045d-2931-4a45-9a5b-89f6a5a78652","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"5693045d-2931-4a45-9a5b-89f6a5a78652","val":4222192738},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":4} 2021-02-08T07:32:53.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:53.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:53.134 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:53.213 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:53.213 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:53.262 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:32:53.250+0000 7f85a0ff9700 -1 failed to decode period 2021-02-08T07:32:53.262 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:32:53.250+0000 7f7541ffb700 -1 failed to decode period 2021-02-08T07:32:53.263 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:32:53.256+0000 7f71d67fc700 -1 failed to decode period 2021-02-08T07:32:53.282 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:32:53.276+0000 7f71ba07b080 0 zone id 5693045d-2931-4a45-9a5b-89f6a5a78652 is not a part of zonegroup test-zonegroup 2021-02-08T07:32:53.283 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:32:53.276+0000 7f71ba07b080 0 zone id 5693045d-2931-4a45-9a5b-89f6a5a78652 is not a part of zonegroup default 2021-02-08T07:32:53.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:53.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:53.297 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:53.380 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:53.381 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:53.433 INFO:teuthology.orchestra.run.smithi017.stderr:2021-02-08T07:32:53.428+0000 7f634e6ef080 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-02-08T07:32:53.440 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-08T07:32:53.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:53.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:53.440 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:53.514 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:53.515 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:53.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:53.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:53.550 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:53.632 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:53.904 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:32:53.890+0000 7f7541ffb700 -1 failed to decode period 2021-02-08T07:32:53.904 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:32:53.890+0000 7f85a0ff9700 -1 failed to decode period 2021-02-08T07:32:53.932 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","epoch":9,"predecessor_uuid":"49be1c29-d1bd-4210-9e84-e35ba7fb03cf","sync_status":["","","","1_1612769180.317275_302.1"],"period_map":{"id":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","zones":[{"id":"4c93366d-9954-4fa2-95e3-91873db27ab5","name":"test-zone4","endpoints":["http://smithi110.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"757db88c-ee56-405d-bd0c-79b9f5b05722","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"c897876e-5513-44a9-a99f-ee5993968e59","name":"test-zone2","endpoints":["http://smithi110.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"4c93366d-9954-4fa2-95e3-91873db27ab5","val":2487017807},{"key":"757db88c-ee56-405d-bd0c-79b9f5b05722","val":1795764096},{"key":"8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","val":2001899811},{"key":"c897876e-5513-44a9-a99f-ee5993968e59","val":696821980}]},"master_zonegroup":"test-zonegroup","master_zone":"757db88c-ee56-405d-bd0c-79b9f5b05722","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"c186a644-2499-4387-8680-a2142bd4cec7","realm_name":"test-realm","realm_epoch":4} 2021-02-08T07:32:53.933 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-02-08T07:32:53.933 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-50 2021-02-08T07:32:54.163 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:32:54.156+0000 7f71d67fc700 -1 failed to decode period 2021-02-08T07:32:54.534 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-51 2021-02-08T07:32:54.637 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-52 2021-02-08T07:32:54.715 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-53 2021-02-08T07:32:54.767 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-54 2021-02-08T07:32:54.838 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-55 2021-02-08T07:32:58.294 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2021-02-08T07:32:58.295 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-56 2021-02-08T07:32:58.887 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-57 2021-02-08T07:32:58.986 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-58 2021-02-08T07:32:59.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'lhddgo-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:59.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'lhddgo-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:59.049 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket lhddgo-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:59.086 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:59.086 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:59.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'lhddgo-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:59.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'lhddgo-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:59.420 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket lhddgo-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:59.500 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:59.501 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:32:59.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'lhddgo-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:59.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'lhddgo-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:32:59.813 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket lhddgo-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:32:59.898 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:32:59.898 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:00.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:00.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:00.241 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:00.321 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:00.322 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:00.621 INFO:teuthology.orchestra.run.smithi017.stdout:["2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44-lhddgo-35","obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41-lhddgo-32","obj_delete_at_hint.0000000104","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-48:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5:8","obj_delete_at_hint.0000000026","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47-lhddgo-38","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5-lhddgo-48","obj_delete_at_hint.0000000028","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:5","datalog.sync-status.c897876e-5513-44a9-a99f-ee5993968e59","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32:8","obj_delete_at_hint.0000000040","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24:1","meta.full-sync.index.0","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:3","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:1","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:3","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-47:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4:8","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3-lhddgo-46","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:3","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2-lhddgo-2","data.full-sync.index.c897876e-5513-44a9-a99f-ee5993968e59.0","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38-lhddgo-29","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:7","obj_delete_at_hint.0000000024","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:3","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31-lhddgo-22","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:5","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27-lhddgo-18","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:5","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22-lhddgo-13","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-48:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5:8","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.3.retry","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-49:757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1:5","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.1","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:3","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5-lhddgo-5","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6-lhddgo-6","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46-lhddgo-37","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.2.retry","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","data.full-sync.index.c897876e-5513-44a9-a99f-ee5993968e59.2","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1-lhddgo-1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:3","bilog.trim","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31:8","obj_delete_at_hint.0000000016","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.2-lhddgo-51","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.1-lhddgo-50","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1-lhddgo-49","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43-lhddgo-34","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37-lhddgo-28","obj_delete_at_hint.0000000014","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35-lhddgo-26","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.39-lhddgo-30","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22:1","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2-lhddgo-45","obj_delete_at_hint.0000000074","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.0.retry","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-46:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3:8","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:7","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40-lhddgo-31","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21-lhddgo-12","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48-lhddgo-39","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:7","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:5","data.full-sync.index.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.2","obj_delete_at_hint.0000000125","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:1","obj_delete_at_hint.0000000082","2021-02-08-07-c897876e-5513-44a9-a99f-ee5993968e59.5488.2-lhddgo-43","meta.full-sync.index.1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33:8","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.3","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-47:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4:8","obj_delete_at_hint.0000000077","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23:1","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16-lhddgo-7","obj_delete_at_hint.0000000032","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.2","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34:8","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47:10","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:1","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.1","mdlog.sync-status.shard.2","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.36-lhddgo-27","obj_delete_at_hint.0000000045","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.0","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33-lhddgo-24","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.6-lhddgo-55","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47:10","mdlog.sync-status.shard.1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-49:757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1:5","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:3","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-46:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46:10","obj_delete_at_hint.0000000066","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45:10","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17-lhddgo-8","obj_delete_at_hint.0000000068","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31:8","obj_delete_at_hint.0000000085","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29-lhddgo-20","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.1.retry","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50-lhddgo-41","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:7","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.7-lhddgo-56","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28-lhddgo-19","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:8","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4-lhddgo-47","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20-lhddgo-11","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22:1","mdlog.sync-status.shard.0","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:3","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.3-lhddgo-52","obj_delete_at_hint.0000000039","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49-lhddgo-40","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46:10","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:3","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:7","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:1","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:1","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:8","obj_delete_at_hint.0000000114","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19-lhddgo-10","data_log.0","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:5","2021-02-08-07-c897876e-5513-44a9-a99f-ee5993968e59.5488.1-lhddgo-42","obj_delete_at_hint.0000000012","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.3.retry","data.full-sync.index.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.3","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","meta.log.38375dfb-4fb1-40a7-81e1-7d3f26b8d208.3","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:5","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3-lhddgo-3","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4-lhddgo-4","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30-lhddgo-21","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:8","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.0","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18-lhddgo-9","obj_delete_at_hint.0000000025","data.full-sync.index.c897876e-5513-44a9-a99f-ee5993968e59.1","obj_delete_at_hint.0000000123","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.2","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.2.retry","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45-lhddgo-36","data.full-sync.index.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.0","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:3","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:3","obj_delete_at_hint.0000000108","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:8","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","obj_delete_at_hint.0000000051","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.0.retry","obj_delete_at_hint.0000000106","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.42-lhddgo-33","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:8","obj_delete_at_hint.0000000017","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:5","obj_delete_at_hint.0000000023","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:3","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:7","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.8-lhddgo-57","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.9-lhddgo-58","meta.full-sync.index.2","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24-lhddgo-15","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:5","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32-lhddgo-23","data_log.2","obj_delete_at_hint.0000000067","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26-lhddgo-17","obj_delete_at_hint.0000000086","datalog.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33:8","data.full-sync.index.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.5-lhddgo-54","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34:8","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.3","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.4-lhddgo-53","obj_delete_at_hint.0000000050","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:1","obj_delete_at_hint.0000000102","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45:10","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","data.full-sync.index.c897876e-5513-44a9-a99f-ee5993968e59.3","obj_delete_at_hint.0000000061","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25-lhddgo-16","obj_delete_at_hint.0000000097","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23-lhddgo-14","obj_delete_at_hint.0000000042","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1-lhddgo-44","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.1.retry","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34-lhddgo-25"] 2021-02-08T07:33:00.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:00.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:00.655 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:00.694 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:00.694 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:00.954 INFO:teuthology.orchestra.run.smithi110.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47-lhddgo-38","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5-lhddgo-48","obj_delete_at_hint.0000000028","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:5","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32:8","obj_delete_at_hint.0000000040","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24:1","meta.full-sync.index.0","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:3","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:1","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:3","obj_delete_at_hint.0000000015","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:3","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:7","obj_delete_at_hint.0000000069","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3-lhddgo-46","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:8","obj_delete_at_hint.0000000035","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:5","obj_delete_at_hint.0000000037","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2-lhddgo-2","obj_delete_at_hint.0000000024","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:3","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31-lhddgo-22","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:5","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27-lhddgo-18","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:1","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.0","obj_delete_at_hint.0000000052","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22-lhddgo-13","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-48:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-49:757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1:5","obj_delete_at_hint.0000000088","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47:10","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:3","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5-lhddgo-5","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6-lhddgo-6","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46-lhddgo-37","obj_delete_at_hint.0000000096","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-46:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3:8","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1-lhddgo-1","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.0","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:3","bilog.trim","obj_delete_at_hint.0000000016","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.2-lhddgo-51","obj_delete_at_hint.0000000041","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.3","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1-lhddgo-49","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.43-lhddgo-34","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.37-lhddgo-28","obj_delete_at_hint.0000000014","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35-lhddgo-26","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22:1","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2-lhddgo-45","obj_delete_at_hint.0000000074","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.0.retry","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-46:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.40-lhddgo-31","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21-lhddgo-12","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48-lhddgo-39","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24:1","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:7","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:5","data.full-sync.index.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.2","obj_delete_at_hint.0000000125","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:1","obj_delete_at_hint.0000000082","2021-02-08-07-c897876e-5513-44a9-a99f-ee5993968e59.5488.2-lhddgo-43","meta.full-sync.index.1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:5","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.3","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-47:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:3","obj_delete_at_hint.0000000077","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16-lhddgo-7","obj_delete_at_hint.0000000032","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.2","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.2.retry","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.1","mdlog.sync-status.shard.2","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:1","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.3","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33-lhddgo-24","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34:8","obj_delete_at_hint.0000000034","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46:10","obj_delete_at_hint.0000000101","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47:10","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:5","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46:10","obj_delete_at_hint.0000000066","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45:10","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17-lhddgo-8","obj_delete_at_hint.0000000068","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31:8","obj_delete_at_hint.0000000085","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29-lhddgo-20","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.1.retry","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50-lhddgo-41","obj_delete_at_hint.0000000073","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23:1","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28-lhddgo-19","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:8","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4-lhddgo-47","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20-lhddgo-11","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49-lhddgo-40","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-48:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:7","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:1","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19-lhddgo-10","data_log.0","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:1","2021-02-08-07-c897876e-5513-44a9-a99f-ee5993968e59.5488.1-lhddgo-42","obj_delete_at_hint.0000000012","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.3.retry","data.full-sync.index.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.3","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","meta.log.38375dfb-4fb1-40a7-81e1-7d3f26b8d208.3","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3-lhddgo-3","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4-lhddgo-4","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30-lhddgo-21","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:8","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.1","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.0","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:5","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.2","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","sync.error-log.1","obj_delete_at_hint.0000000033","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18-lhddgo-9","obj_delete_at_hint.0000000025","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:3","obj_delete_at_hint.0000000123","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29:8","datalog.sync-status.shard.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.2.retry","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45-lhddgo-36","data.full-sync.index.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.0","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:3","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45:10","obj_delete_at_hint.0000000060","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31:8","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","obj_delete_at_hint.0000000051","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:3","obj_delete_at_hint.0000000106","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:8","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:7","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.2","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.8-lhddgo-57","meta.full-sync.index.2","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24-lhddgo-15","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:5","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-49:757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1:5","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32-lhddgo-23","data_log.2","obj_delete_at_hint.0000000067","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26-lhddgo-17","obj_delete_at_hint.0000000086","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:3","datalog.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33:8","data.full-sync.index.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d.1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:7","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.5-lhddgo-54","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.0.retry","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:8","bucket.sync-status.8e9c5146-39d0-40f8-b41d-7c27e66f6e3d:lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:3","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.3.retry","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-47:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:7","obj_delete_at_hint.0000000001","datalog.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25-lhddgo-16","obj_delete_at_hint.0000000097","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23-lhddgo-14","obj_delete_at_hint.0000000042","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1-lhddgo-44","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:1","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.1.retry","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34-lhddgo-25"] 2021-02-08T07:33:00.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:00.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:00.989 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:01.028 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:01.028 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:01.309 INFO:teuthology.orchestra.run.smithi017.stdout:["2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.44-lhddgo-35","obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.41-lhddgo-32","obj_delete_at_hint.0000000104","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-48:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5:8","obj_delete_at_hint.0000000026","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47-lhddgo-38","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5-lhddgo-48","obj_delete_at_hint.0000000028","datalog.sync-status.c897876e-5513-44a9-a99f-ee5993968e59","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-47:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4:8","obj_delete_at_hint.0000000015","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:3","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:7","obj_delete_at_hint.0000000069","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3-lhddgo-46","obj_delete_at_hint.0000000095","meta.log.49be1c29-d1bd-4210-9e84-e35ba7fb03cf.3","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:8","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:3","obj_delete_at_hint.0000000035","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:5","obj_delete_at_hint.0000000037","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.2-lhddgo-2","data.full-sync.index.c897876e-5513-44a9-a99f-ee5993968e59.0","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.38-lhddgo-29","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:7","obj_delete_at_hint.0000000024","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31-lhddgo-22","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32:8","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27-lhddgo-18","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:5","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:1","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.0","obj_delete_at_hint.0000000052","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22-lhddgo-13","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33:8","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.3.retry","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.1","obj_delete_at_hint.0000000088","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47:10","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.5-lhddgo-5","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.6-lhddgo-6","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46-lhddgo-37","obj_delete_at_hint.0000000096","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-46:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3:8","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.2.retry","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","data.full-sync.index.c897876e-5513-44a9-a99f-ee5993968e59.2","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.1-lhddgo-1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:8","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.0","bilog.trim","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31:8","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.3","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1-lhddgo-49","obj_delete_at_hint.0000000014","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35-lhddgo-26","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.2-lhddgo-45","obj_delete_at_hint.0000000074","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:8","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:7","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21-lhddgo-12","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.48-lhddgo-39","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24:1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","2021-02-08-07-c897876e-5513-44a9-a99f-ee5993968e59.5488.2-lhddgo-43","meta.full-sync.index.1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:5","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:3","obj_delete_at_hint.0000000077","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.16-lhddgo-7","obj_delete_at_hint.0000000032","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34:8","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.2.retry","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47:10","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:1","mdlog.sync-status.shard.2","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.3","obj_delete_at_hint.0000000045","meta.log.d4b11cd5-1a7c-4383-afcf-402844a1791d.0","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.0","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33-lhddgo-24","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.6-lhddgo-55","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34:8","obj_delete_at_hint.0000000034","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46:10","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-49:757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1:5","meta.log.d4b11cd5-1a7c-4383-afcf-402844a1791d.2","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:5","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35:8","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:3","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-46:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3:8","obj_delete_at_hint.0000000066","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.17-lhddgo-8","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29-lhddgo-20","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.50-lhddgo-41","obj_delete_at_hint.0000000073","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23:1","obj_delete_at_hint.0000000038","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:7","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28-lhddgo-19","meta.log.d4b11cd5-1a7c-4383-afcf-402844a1791d.3","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4-lhddgo-47","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20-lhddgo-11","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22:1","mdlog.sync-status.shard.0","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:3","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.3-lhddgo-52","obj_delete_at_hint.0000000039","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.49-lhddgo-40","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46:10","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-48:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5:8","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:3","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:1","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:8","obj_delete_at_hint.0000000114","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19-lhddgo-10","data_log.0","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:5","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:1","2021-02-08-07-c897876e-5513-44a9-a99f-ee5993968e59.5488.1-lhddgo-42","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","meta.log.38375dfb-4fb1-40a7-81e1-7d3f26b8d208.3","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:5","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.3-lhddgo-3","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.4-lhddgo-4","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30-lhddgo-21","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:5","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.2","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:1","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.18-lhddgo-9","obj_delete_at_hint.0000000025","data.full-sync.index.c897876e-5513-44a9-a99f-ee5993968e59.1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:3","obj_delete_at_hint.0000000123","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29:8","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.2","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45-lhddgo-36","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:3","obj_delete_at_hint.0000000108","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:8","obj_delete_at_hint.0000000094","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45:10","obj_delete_at_hint.0000000060","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31:8","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","obj_delete_at_hint.0000000051","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:3","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.0.retry","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:5","obj_delete_at_hint.0000000023","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:3","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.2","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.6124.9-lhddgo-58","meta.full-sync.index.2","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24-lhddgo-15","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-49:757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1:5","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32-lhddgo-23","data_log.2","obj_delete_at_hint.0000000067","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26-lhddgo-17","obj_delete_at_hint.0000000086","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:3","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:7","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.0.retry","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:8","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.3","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:3","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.3.retry","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-47:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4:8","obj_delete_at_hint.0000000050","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:1","obj_delete_at_hint.0000000102","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45:10","obj_delete_at_hint.0000000124","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:7","obj_delete_at_hint.0000000001","datalog.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","data.full-sync.index.c897876e-5513-44a9-a99f-ee5993968e59.3","obj_delete_at_hint.0000000061","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25-lhddgo-16","obj_delete_at_hint.0000000097","bucket.sync-status.c897876e-5513-44a9-a99f-ee5993968e59:lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32:8","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23-lhddgo-14","obj_delete_at_hint.0000000042","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.5803.1-lhddgo-44","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:1","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.1.retry","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","datalog.sync-status.shard.c897876e-5513-44a9-a99f-ee5993968e59.1.retry","2021-02-08-07-757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34-lhddgo-25"] 2021-02-08T07:33:01.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:01.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:01.345 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:01.383 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:01.383 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:01.618 INFO:teuthology.orchestra.run.smithi110.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:3","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:7","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","meta.log.49be1c29-d1bd-4210-9e84-e35ba7fb03cf.3","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:8","obj_delete_at_hint.0000000035","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:5","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-23:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.32:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-13:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.22:1","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:1","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.0","obj_delete_at_hint.0000000052","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-24:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.33:8","obj_delete_at_hint.0000000088","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-38:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.47:10","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-46:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.3:8","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.0","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.3","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","obj_delete_at_hint.0000000014","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-21:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.30:8","obj_delete_at_hint.0000000074","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:8","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-15:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.24:1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:5","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-10:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.19:3","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.2.retry","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","mdlog.sync-status.shard.2","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.3","obj_delete_at_hint.0000000045","meta.log.d4b11cd5-1a7c-4383-afcf-402844a1791d.0","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-25:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.34:8","obj_delete_at_hint.0000000034","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-37:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.46:10","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","meta.log.d4b11cd5-1a7c-4383-afcf-402844a1791d.2","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-19:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.28:5","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-26:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.35:8","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-14:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.23:1","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","meta.log.d4b11cd5-1a7c-4383-afcf-402844a1791d.3","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-48:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.5:8","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:1","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","meta.log.38375dfb-4fb1-40a7-81e1-7d3f26b8d208.3","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.1","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-18:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.27:5","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.2","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-17:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.26:3","obj_delete_at_hint.0000000123","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-20:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.29:8","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-36:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.45:10","obj_delete_at_hint.0000000060","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-22:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.31:8","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","obj_delete_at_hint.0000000051","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:3","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","data.full-sync.index.757db88c-ee56-405d-bd0c-79b9f5b05722.2","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-49:757db88c-ee56-405d-bd0c-79b9f5b05722.5991.1:5","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-16:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.25:3","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:7","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.0.retry","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:8","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:3","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.3.retry","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-47:757db88c-ee56-405d-bd0c-79b9f5b05722.5803.4:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-12:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.21:7","obj_delete_at_hint.0000000001","datalog.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.757db88c-ee56-405d-bd0c-79b9f5b05722:lhddgo-11:757db88c-ee56-405d-bd0c-79b9f5b05722.4347.20:1","datalog.sync-status.shard.757db88c-ee56-405d-bd0c-79b9f5b05722.1.retry","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2021-02-08T07:33:01.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:01.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:01.658 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:01.698 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:01.698 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:01.971 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769580.200601_290.3","last_update":"2021-02-08T07:33:00.200601Z"},{"marker":"1_1612769580.200601_626.2","last_update":"2021-02-08T07:33:00.200601Z"},{"marker":"1_1612769580.200601_481.3","last_update":"2021-02-08T07:33:00.200601Z"},{"marker":"1_1612769580.200601_392.3","last_update":"2021-02-08T07:33:00.200601Z"}] 2021-02-08T07:33:01.972 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-08T07:33:01.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:01.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:01.972 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:02.016 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:02.016 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:02.272 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:02.272 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:07.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:07.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:07.274 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:07.316 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:07.317 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:07.553 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:07.554 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:12.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:12.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:12.555 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:12.598 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:12.598 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:12.833 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:12.834 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:17.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:17.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:17.836 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:18.120 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:18.121 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:23.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:23.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:23.123 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:23.418 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:23.419 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:28.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:28.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:28.420 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:28.462 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:28.463 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:28.695 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:28.695 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:33.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:33.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:33.697 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:33.739 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:33.739 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:33.977 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:33.977 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:38.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:38.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:38.979 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:39.022 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:39.022 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:39.265 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:39.265 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:44.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:44.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:44.267 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:44.310 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:44.310 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:44.558 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:44.559 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:49.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:49.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:49.560 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:49.602 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:49.602 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:49.843 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:49.844 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:54.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:54.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:33:54.846 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:33:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:33:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:33:55.130 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.395761_173.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.395761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768880.006161_251.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:20.006161Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.789494_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.789494Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769440.138962_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:30:40.138962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:33:55.130 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1612769580.200601_290.3 target=1_1612769107.395761_173.1, shard=1 master=1_1612769580.200601_626.2 target=1_1612768880.006161_251.3, shard=2 master=1_1612769580.200601_481.3 target=1_1612769271.789494_416.1, shard=3 master=1_1612769580.200601_392.3 target=1_1612769440.138962_328.1 2021-02-08T07:33:55.542 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72c5ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.542 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72c2fed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.542 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72dd7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.542 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72deffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.542 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72c97fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.544 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72c6ff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.544 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72cbfff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.544 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72c3fef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.544 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72c37ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.544 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72df7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.544 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72d37fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.544 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72c37ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.545 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72caffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.545 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72c77f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.545 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72c77f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.545 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.535+0000 7f72dd7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.545 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.539+0000 7f72deffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:55.545 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2021-02-08T07:33:55.539+0000 7f72d3fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.187 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f86977fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.187 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f868c7f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.187 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f86967fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.187 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f86957fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.187 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f8695ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.187 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f868f7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.187 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f868effd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.187 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f868e7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f8689ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f868ffff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f86867ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f8688ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f868a7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f86887f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f868aff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f86857ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f8697fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f86ad7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f86897f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f8687fef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.188 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.173+0000 7f8686fed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.189 INFO:tasks.rgw.c2.client.0.smithi110.stdout:2021-02-08T07:33:57.177+0000 7f86977fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.324 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.313+0000 7f76337f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.324 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.313+0000 7f7632ff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.324 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.313+0000 7f7634ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.325 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.313+0000 7f76317f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.325 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.313+0000 7f7633ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.325 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.313+0000 7f763d7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.325 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.313+0000 7f76327f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.325 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.313+0000 7f76357fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.325 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.313+0000 7f76347f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.328 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f763f7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.328 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f762ffef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f76307f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f7632ff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f762efed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f7634ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f76317f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f763e7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f7633ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f763d7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f76327f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.329 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f763effd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.330 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f763dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.330 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f76377fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.330 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f764affd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.330 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f7631ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.330 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f762f7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.331 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f762e7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.331 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f7648ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.331 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f7630ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.331 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f7649ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.331 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.317+0000 7f763ffff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.331 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.321+0000 7f76497fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:33:57.332 INFO:tasks.rgw.c2.client.1.smithi110.stdout:2021-02-08T07:33:57.321+0000 7f764a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-08T07:34:00.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:00.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:00.132 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:00.415 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_290.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_626.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_481.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_392.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:00.416 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-08T07:34:00.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:00.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:00.416 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:00.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:00.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:00.738 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769580.200601_290.3","last_update":"2021-02-08T07:33:00.200601Z"},{"marker":"1_1612769580.200601_626.2","last_update":"2021-02-08T07:33:00.200601Z"},{"marker":"1_1612769580.200601_481.3","last_update":"2021-02-08T07:33:00.200601Z"},{"marker":"1_1612769580.200601_392.3","last_update":"2021-02-08T07:33:00.200601Z"}] 2021-02-08T07:34:00.738 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-08T07:34:00.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:00.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:00.739 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:00.835 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:00.835 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:01.072 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7430472880135713257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_290.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_626.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_481.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_392.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:01.073 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-08T07:34:01.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:01.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:01.073 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:01.110 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:01.110 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:01.373 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769580.200601_290.3","last_update":"2021-02-08T07:33:00.200601Z"},{"marker":"1_1612769580.200601_626.2","last_update":"2021-02-08T07:33:00.200601Z"},{"marker":"1_1612769580.200601_481.3","last_update":"2021-02-08T07:33:00.200601Z"},{"marker":"1_1612769580.200601_392.3","last_update":"2021-02-08T07:33:00.200601Z"}] 2021-02-08T07:34:01.373 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-08T07:34:01.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:01.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:01.374 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:01.415 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:01.415 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:01.693 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":729464676215090460},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_290.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_626.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_481.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769580.200601_392.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:00.200601Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:01.693 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-08T07:34:01.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:01.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:01.694 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:01.772 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:01.772 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:02.011 INFO:teuthology.orchestra.run.smithi110.stdout:[{"marker":"1_1612769641.156483_216.1","last_update":"2021-02-08T07:34:01.156483Z"},{"marker":"1_1612769641.150011_631.1","last_update":"2021-02-08T07:34:01.150011Z"},{"marker":"1_1612769641.152199_496.1","last_update":"2021-02-08T07:34:01.152199Z"},{"marker":"1_1612769641.154292_406.1","last_update":"2021-02-08T07:34:01.154292Z"}] 2021-02-08T07:34:02.011 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-08T07:34:02.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:02.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:02.012 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:02.056 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:02.056 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:02.326 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:02.327 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:07.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:07.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:07.329 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:07.372 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:07.372 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:07.637 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:07.638 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:12.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:12.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:12.639 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:12.675 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:12.675 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:12.938 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:12.939 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:17.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:17.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:17.939 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:17.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:17.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:18.260 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:18.260 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:23.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:23.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:23.261 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:23.304 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:23.305 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:23.583 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:23.583 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:28.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:28.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:28.585 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:28.631 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:28.631 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:28.900 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:28.901 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:33.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:33.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:33.902 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:33.946 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:33.947 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:34.225 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:34.225 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:39.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:39.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:39.227 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:39.269 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:39.269 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:39.535 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:39.535 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:44.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:44.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:44.536 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:44.579 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:44.579 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:44.847 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:44.847 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:49.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:49.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:49.848 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:49.891 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:49.891 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:50.144 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:50.145 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:34:55.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:55.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:34:55.146 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:34:55.188 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:34:55.189 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:34:55.447 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769107.392980_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:25:07.392980Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612768881.007212_292.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:21:21.007212Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769271.739830_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:27:51.739830Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769463.219895_357.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:31:03.219895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:34:55.447 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1612769641.156483_216.1 target=1_1612769107.392980_109.1, shard=1 master=1_1612769641.150011_631.1 target=1_1612768881.007212_292.3, shard=2 master=1_1612769641.152199_496.1 target=1_1612769271.739830_424.1, shard=3 master=1_1612769641.154292_406.1 target=1_1612769463.219895_357.1 2021-02-08T07:35:00.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:00.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:00.448 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:00.491 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:00.491 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:00.771 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769641.156483_216.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:34:01.156483Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769641.150011_631.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:34:01.150011Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769641.152199_496.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:34:01.152199Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769641.154292_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:34:01.154292Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:35:00.771 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-08T07:35:00.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:00.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:00.771 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:00.811 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:00.812 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:01.056 INFO:teuthology.orchestra.run.smithi110.stdout:[{"marker":"1_1612769641.156483_216.1","last_update":"2021-02-08T07:34:01.156483Z"},{"marker":"1_1612769641.150011_631.1","last_update":"2021-02-08T07:34:01.150011Z"},{"marker":"1_1612769641.152199_496.1","last_update":"2021-02-08T07:34:01.152199Z"},{"marker":"1_1612769641.154292_406.1","last_update":"2021-02-08T07:34:01.154292Z"}] 2021-02-08T07:35:01.056 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-08T07:35:01.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:01.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:01.057 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:01.101 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:01.101 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:01.370 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":711522086589135958},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769641.156483_216.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:34:01.156483Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769641.150011_631.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:34:01.150011Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769641.152199_496.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:34:01.152199Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769641.154292_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:34:01.154292Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:35:01.371 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-08T07:35:01.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:01.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:01.371 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:01.451 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:01.715 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769639.724633_204.1","last_update":"2021-02-08T07:33:59.724633Z"},{"marker":"1_1612769639.717934_480.1","last_update":"2021-02-08T07:33:59.717934Z"},{"marker":"1_1612769639.720157_400.1","last_update":"2021-02-08T07:33:59.720157Z"},{"marker":"1_1612769639.722373_279.1","last_update":"2021-02-08T07:33:59.722373Z"}] 2021-02-08T07:35:01.716 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-08T07:35:01.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:01.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:01.716 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:01.802 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:01.802 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:02.062 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12652603110286941624},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769639.724633_204.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.724633Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769639.717934_480.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.717934Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769639.720157_400.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.720157Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769639.722373_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.722373Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:35:02.062 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-08T07:35:02.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:02.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:02.063 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:02.143 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:02.143 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:02.408 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769639.724633_204.1","last_update":"2021-02-08T07:33:59.724633Z"},{"marker":"1_1612769639.717934_480.1","last_update":"2021-02-08T07:33:59.717934Z"},{"marker":"1_1612769639.720157_400.1","last_update":"2021-02-08T07:33:59.720157Z"},{"marker":"1_1612769639.722373_279.1","last_update":"2021-02-08T07:33:59.722373Z"}] 2021-02-08T07:35:02.408 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-08T07:35:02.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:02.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:02.409 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:02.452 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:02.690 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5378454525578809946},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769639.724633_204.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.724633Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769639.717934_480.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.717934Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769639.720157_400.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.720157Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769639.722373_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.722373Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:35:02.690 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-08T07:35:02.691 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2021-02-08T07:35:02.692 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-59 2021-02-08T07:35:02.737 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-60 2021-02-08T07:35:02.816 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-61 2021-02-08T07:35:03.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:03.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:03.511 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:03.559 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:03.559 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:03.802 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.60.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:03.802 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:35:03.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:03.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:03.803 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:03.844 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:03.845 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:04.076 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:04.076 WARNING:rgw_multi.tests:bucket lhddgo-59 zone test-zone2 behind zone test-zone1: shard=2 master=00000000001.37.5 target=, shard=7 master=00000000001.60.5 target= 2021-02-08T07:35:09.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:09.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:09.077 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:09.122 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:09.122 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:09.378 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:09.378 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:35:09.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:09.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:09.379 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:09.420 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:09.420 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:09.700 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.60.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:09.700 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:35:09.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:09.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:09.701 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:09.790 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:09.790 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:10.074 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:10.074 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:35:10.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:10.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:10.075 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:10.155 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:10.155 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:10.427 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.60.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:10.427 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:35:10.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:10.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:10.428 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:10.841 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.37.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.60.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:10.842 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:35:10.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:10.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:10.842 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:10.917 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:10.917 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:11.153 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.71.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.35.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.48.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.65.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:11.153 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-59 2021-02-08T07:35:11.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:11.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:11.154 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:11.200 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:11.200 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:11.503 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:11.503 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-59 2021-02-08T07:35:11.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:11.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:11.504 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:11.769 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.71.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.35.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.48.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.65.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:11.770 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-59 2021-02-08T07:35:11.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:11.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:11.771 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:11.815 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:11.816 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:12.100 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:12.101 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-59 2021-02-08T07:35:12.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:12.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:12.101 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:12.191 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:12.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:12.458 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.41.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.84.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:12.458 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-59 2021-02-08T07:35:12.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:12.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:12.459 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:12.549 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:12.549 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:12.825 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.84.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:12.825 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-59 2021-02-08T07:35:12.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:12.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:12.826 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:12.911 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:12.911 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:13.172 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.41.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.84.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:13.174 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-59 2021-02-08T07:35:13.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:13.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:13.178 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:13.227 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:13.227 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:13.494 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.84.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:13.494 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-59 2021-02-08T07:35:13.494 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-59 zones={test-zone2, test-zone1} 2021-02-08T07:35:14.367 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:35:14.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:14.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:14.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:14.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:14.375 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:35:14.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:14.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:14.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:14.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:14.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:14.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:14.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:14.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:14.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:14.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:14.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:15.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:15.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:15.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:15.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:15.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:15.525 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-59 zones={test-zone2, test-zone1} 2021-02-08T07:35:15.525 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-59 zones={test-zone3, test-zone1} 2021-02-08T07:35:15.739 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:35:15.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:15.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:15.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:15.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:15.745 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:35:15.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:15.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:15.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:15.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:15.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:15.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:15.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:16.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:16.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:16.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:16.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:16.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:16.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:16.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:16.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:16.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:16.891 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-59 zones={test-zone3, test-zone1} 2021-02-08T07:35:16.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-59 zones={test-zone3, test-zone2} 2021-02-08T07:35:17.449 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:35:17.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:17.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:17.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:17.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:17.458 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:35:17.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:17.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:17.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:17.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:17.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:17.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:17.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:17.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:17.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:17.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:18.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:18.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:18.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:18.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:18.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:18.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:18.606 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-59 zones={test-zone3, test-zone2} 2021-02-08T07:35:18.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:18.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:18.606 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:18.650 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:18.651 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:18.927 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.201.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.36.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.81.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.46.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:18.927 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:35:18.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:18.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:18.927 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:18.971 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:18.971 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:19.244 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.201.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:19.244 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:35:19.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:19.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:19.245 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:19.289 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:19.290 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:19.557 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.201.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.36.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.81.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.46.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:19.557 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:35:19.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:19.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:19.558 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:19.649 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:19.649 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:19.935 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.201.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:19.935 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:35:19.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:19.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:19.936 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:19.977 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:19.977 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:20.248 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.201.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.36.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.81.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.46.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:20.249 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:35:20.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:20.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:20.250 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:20.294 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:20.294 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:20.568 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.201.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.36.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.81.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.46.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:20.569 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:35:20.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:20.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:20.569 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:20.607 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:20.607 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:20.848 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.254.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.71.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:20.848 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-60 2021-02-08T07:35:20.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:20.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:20.849 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:20.893 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:20.894 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:21.201 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.254.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:21.201 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-60 2021-02-08T07:35:21.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:21.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:21.202 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:21.243 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:21.243 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:21.488 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.254.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.71.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:21.489 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-60 2021-02-08T07:35:21.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:21.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:21.489 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:21.542 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:21.542 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:21.807 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.254.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:21.807 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-60 2021-02-08T07:35:21.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:21.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:21.808 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:21.848 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:21.848 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:22.137 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.218.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.67.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:22.137 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-60 2021-02-08T07:35:22.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:22.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:22.138 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:22.223 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:22.223 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:22.537 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.218.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:22.537 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-60 2021-02-08T07:35:22.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:22.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:22.538 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:22.619 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:22.620 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:22.912 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.218.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.67.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:22.913 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-60 2021-02-08T07:35:22.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:22.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:22.914 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:22.960 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:22.960 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:23.232 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.218.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:23.233 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-60 2021-02-08T07:35:23.233 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-60 zones={test-zone2, test-zone1} 2021-02-08T07:35:24.988 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:35:24.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:24.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:24.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:24.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:24.996 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:35:25.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:25.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:25.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:25.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:25.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:25.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:25.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:25.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:25.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:25.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:25.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:25.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:25.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:25.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:25.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:26.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:26.144 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-60 zones={test-zone2, test-zone1} 2021-02-08T07:35:26.145 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-60 zones={test-zone3, test-zone1} 2021-02-08T07:35:26.873 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:35:26.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:26.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:26.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:26.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:26.880 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:35:26.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:26.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:26.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:26.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:26.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:26.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:27.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:27.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:27.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:27.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:27.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:27.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:27.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:27.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:27.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:27.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:28.031 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-60 zones={test-zone3, test-zone1} 2021-02-08T07:35:28.032 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-60 zones={test-zone3, test-zone2} 2021-02-08T07:35:28.881 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:35:28.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:28.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:28.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:28.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:28.893 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:35:28.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:28.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:28.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:28.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:28.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:28.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:29.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:29.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:29.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:29.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:29.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:29.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:29.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:29.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:29.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:29.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:30.102 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-60 zones={test-zone3, test-zone2} 2021-02-08T07:35:30.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:30.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:30.102 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:30.146 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:30.146 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:30.434 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.40.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.203.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:30.434 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:35:30.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:30.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:30.435 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:30.480 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:30.480 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:30.731 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.203.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:30.731 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:35:30.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:30.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:30.732 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:30.778 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:31.040 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.40.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.203.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:31.040 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:35:31.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:31.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:31.041 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:31.128 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:31.128 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:31.425 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.203.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:31.425 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:35:31.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:31.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:31.426 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:31.466 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:31.466 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:31.762 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.40.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.203.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:31.762 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:35:31.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:31.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:31.763 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:31.806 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:31.806 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:32.089 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.61.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.40.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.203.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:32.089 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:35:32.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:32.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:32.090 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:32.416 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.73.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.38.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.258.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:32.417 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-61 2021-02-08T07:35:32.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:32.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:32.417 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:32.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:32.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:32.753 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.258.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:32.753 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-61 2021-02-08T07:35:32.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:32.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:32.754 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:33.041 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.73.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.38.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.258.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:33.041 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-61 2021-02-08T07:35:33.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:33.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:33.042 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:33.087 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:33.087 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:33.378 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.258.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:33.378 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-61 2021-02-08T07:35:33.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:33.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:33.379 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:33.459 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:33.460 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:33.717 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.80.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.214.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:33.718 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-61 2021-02-08T07:35:33.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:33.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:33.718 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:33.806 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:34.114 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.214.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:34.115 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-61 2021-02-08T07:35:34.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:34.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:34.116 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:34.155 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:34.156 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:34.449 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.80.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.214.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:35:34.449 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-61 2021-02-08T07:35:34.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:34.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:34.450 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:34.497 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:34.497 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:34.752 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.214.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:35:34.753 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-61 2021-02-08T07:35:34.753 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-61 zones={test-zone2, test-zone1} 2021-02-08T07:35:36.067 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:35:36.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:36.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:36.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:36.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:36.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:35:36.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:36.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:36.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:36.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:36.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:36.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:36.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:36.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:36.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:36.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:36.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:36.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:36.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:36.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:37.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:37.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:37.260 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-61 zones={test-zone2, test-zone1} 2021-02-08T07:35:37.260 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-61 zones={test-zone3, test-zone1} 2021-02-08T07:35:37.588 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:35:37.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:37.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:37.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:37.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:37.595 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:35:37.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:37.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:37.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:37.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:37.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:37.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:37.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:37.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:37.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:38.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:38.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:38.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:38.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:38.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:38.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:38.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:38.775 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-61 zones={test-zone3, test-zone1} 2021-02-08T07:35:38.775 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-61 zones={test-zone3, test-zone2} 2021-02-08T07:35:39.229 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:35:39.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:39.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:39.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:39.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:39.239 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:35:39.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:35:39.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:35:39.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:35:39.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:35:39.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:39.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:39.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:35:39.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:39.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:39.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:35:39.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:39.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:39.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:35:40.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:40.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:40.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:35:40.374 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-61 zones={test-zone3, test-zone2} 2021-02-08T07:35:40.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'lhddgo-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:40.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'lhddgo-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:40.374 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket lhddgo-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:35:40.416 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:40.417 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:40.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'lhddgo-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:40.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'lhddgo-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:40.787 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket lhddgo-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:35:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:41.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'lhddgo-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:41.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'lhddgo-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:41.218 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket lhddgo-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:35:41.303 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:41.303 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:41.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'lhddgo-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:41.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'lhddgo-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:41.661 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket lhddgo-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:35:41.746 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:41.746 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:42.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'lhddgo-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:42.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'lhddgo-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:42.109 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket lhddgo-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:35:42.196 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:42.196 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:42.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'lhddgo-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:42.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'lhddgo-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:35:42.560 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket lhddgo-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:35:42.600 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:42.600 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:44.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:44.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:44.413 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:44.469 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:44.469 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:44.806 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.71.5"},{"key":1,"val":"00000000004.40.1"},{"key":2,"val":"00000000006.47.5"},{"key":3,"val":"00000000004.85.1"},{"key":4,"val":"00000000006.55.5"},{"key":5,"val":"00000000004.66.1"},{"key":6,"val":"00000000004.34.1"},{"key":7,"val":"00000000006.72.5"},{"key":8,"val":"00000000004.63.1"},{"key":9,"val":"00000000004.49.1"},{"key":10,"val":"00000000004.50.1"}]} 2021-02-08T07:35:44.807 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:35:44.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:44.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:44.808 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:44.852 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:44.852 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:45.072 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.71.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.47.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.55.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.34.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.72.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}}] 2021-02-08T07:35:45.072 WARNING:rgw_multi.tests:bucket lhddgo-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.40.1 target=, shard=5 master=00000000004.66.1 target=, shard=9 master=00000000004.49.1 target= 2021-02-08T07:35:50.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:50.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:50.073 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:50.114 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:50.366 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.71.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.47.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.55.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.34.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.72.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}}] 2021-02-08T07:35:50.366 WARNING:rgw_multi.tests:bucket lhddgo-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.40.1 target=, shard=5 master=00000000004.66.1 target=, shard=9 master=00000000004.49.1 target= 2021-02-08T07:35:55.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:55.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:35:55.369 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:35:55.413 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:35:55.414 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:35:55.693 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.71.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.47.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.55.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.34.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.72.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}}] 2021-02-08T07:35:55.694 WARNING:rgw_multi.tests:bucket lhddgo-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.40.1 target=, shard=5 master=00000000004.66.1 target=, shard=9 master=00000000004.49.1 target= 2021-02-08T07:36:00.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:00.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:00.695 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:00.737 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:00.737 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:00.989 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.71.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.47.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.55.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.34.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.72.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}}] 2021-02-08T07:36:00.989 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:36:00.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:00.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:00.990 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:01.032 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:01.032 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:01.303 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.71.5"},{"key":1,"val":"00000000004.40.1"},{"key":2,"val":"00000000006.47.5"},{"key":3,"val":"00000000004.85.1"},{"key":4,"val":"00000000006.55.5"},{"key":5,"val":"00000000004.66.1"},{"key":6,"val":"00000000004.34.1"},{"key":7,"val":"00000000006.72.5"},{"key":8,"val":"00000000004.63.1"},{"key":9,"val":"00000000004.49.1"},{"key":10,"val":"00000000004.50.1"}]} 2021-02-08T07:36:01.303 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:36:01.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:01.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:01.304 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:01.393 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:01.394 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:01.682 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.71.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.47.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.55.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.34.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.72.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}}] 2021-02-08T07:36:01.682 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:36:01.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:01.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:01.683 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:01.763 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:01.763 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:02.047 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.71.5"},{"key":1,"val":"00000000004.40.1"},{"key":2,"val":"00000000006.47.5"},{"key":3,"val":"00000000004.85.1"},{"key":4,"val":"00000000006.55.5"},{"key":5,"val":"00000000004.66.1"},{"key":6,"val":"00000000004.34.1"},{"key":7,"val":"00000000006.72.5"},{"key":8,"val":"00000000004.63.1"},{"key":9,"val":"00000000004.49.1"},{"key":10,"val":"00000000004.50.1"}]} 2021-02-08T07:36:02.047 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:36:02.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:02.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:02.048 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:02.092 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:02.092 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:02.369 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.71.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.47.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.55.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.34.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.72.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}}] 2021-02-08T07:36:02.370 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-59 2021-02-08T07:36:02.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:02.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:02.370 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:02.690 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000004.81.5"},{"key":1,"val":"00000000002.48.1"},{"key":2,"val":"00000000004.43.5"},{"key":3,"val":"00000000002.75.1"},{"key":4,"val":"00000000004.56.5"},{"key":5,"val":"00000000002.86.1"},{"key":6,"val":"00000000002.51.1"},{"key":7,"val":"00000000004.75.5"},{"key":8,"val":"00000000002.85.1"},{"key":9,"val":"00000000002.51.1"},{"key":10,"val":"00000000002.55.1"}]} 2021-02-08T07:36:02.690 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-59 2021-02-08T07:36:02.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:02.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:02.691 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:02.736 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:02.736 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:03.014 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.81.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.43.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.56.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.75.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}}] 2021-02-08T07:36:03.015 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-59 2021-02-08T07:36:03.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:03.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:03.016 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:03.057 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:03.057 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:03.308 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000004.81.5"},{"key":1,"val":"00000000002.48.1"},{"key":2,"val":"00000000004.43.5"},{"key":3,"val":"00000000002.75.1"},{"key":4,"val":"00000000004.56.5"},{"key":5,"val":"00000000002.86.1"},{"key":6,"val":"00000000002.51.1"},{"key":7,"val":"00000000004.75.5"},{"key":8,"val":"00000000002.85.1"},{"key":9,"val":"00000000002.51.1"},{"key":10,"val":"00000000002.55.1"}]} 2021-02-08T07:36:03.308 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-59 2021-02-08T07:36:03.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:03.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:03.308 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:03.354 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:03.651 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.81.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.43.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.56.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.75.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}}] 2021-02-08T07:36:03.651 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-59 2021-02-08T07:36:03.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:03.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:03.652 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:04.002 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000002.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.43.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.92.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:04.003 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-59 2021-02-08T07:36:04.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:04.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:04.003 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:04.088 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:04.088 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:04.385 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.63.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.43.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.92.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:04.386 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-59 2021-02-08T07:36:04.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:04.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:04.386 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:04.426 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:04.426 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:04.680 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000002.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.43.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.92.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:04.680 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-59 2021-02-08T07:36:04.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:04.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:04.681 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:04.724 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:04.724 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:04.985 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.63.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.43.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.92.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:04.985 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-59 2021-02-08T07:36:04.985 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-59 zones={test-zone2, test-zone1} 2021-02-08T07:36:05.941 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:05.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:05.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:05.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:05.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:05.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:05.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:05.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:05.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:05.950 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:05.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:05.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:05.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:05.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:05.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:05.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:05.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:05.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:05.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:06.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:06.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:06.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:06.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:06.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:06.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:06.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:06.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:06.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:06.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:06.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:07.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:07.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:07.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:07.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:07.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:07.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:07.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:07.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:07.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:08.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:08.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:08.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:08.304 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-59 zones={test-zone2, test-zone1} 2021-02-08T07:36:08.304 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-59 zones={test-zone3, test-zone1} 2021-02-08T07:36:09.230 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:09.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:09.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:09.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:09.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:09.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:09.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:09.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:09.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:09.239 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:09.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:09.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:09.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:09.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:09.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:09.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:09.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:09.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:09.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:09.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:09.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:10.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:10.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:10.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:10.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:10.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:10.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:10.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:10.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:10.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:11.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:11.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:11.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:11.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:11.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:11.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:11.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:11.807 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-59 zones={test-zone3, test-zone1} 2021-02-08T07:36:11.808 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-59 zones={test-zone3, test-zone2} 2021-02-08T07:36:12.098 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:12.117 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:12.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:12.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:12.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:12.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:12.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:12.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:12.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:12.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:12.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:12.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:12.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:12.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:12.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:12.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:12.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:12.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:12.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:12.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:13.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:13.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:13.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:13.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:13.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:13.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:13.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:13.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:13.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:13.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:14.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:14.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:14.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:14.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:14.421 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-59 zones={test-zone3, test-zone2} 2021-02-08T07:36:14.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:14.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:14.422 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:14.464 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:14.464 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:14.739 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.213.5"},{"key":1,"val":"00000000004.67.1"},{"key":2,"val":"00000000006.46.5"},{"key":3,"val":"00000000004.70.1"},{"key":4,"val":"00000000006.91.5"},{"key":5,"val":"00000000004.39.1"},{"key":6,"val":"00000000004.44.1"},{"key":7,"val":"00000000006.56.5"},{"key":8,"val":"00000000004.54.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.53.1"}]} 2021-02-08T07:36:14.740 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:36:14.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:14.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:14.740 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:14.783 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:14.783 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:15.035 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.213.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.46.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.91.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.56.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}}] 2021-02-08T07:36:15.036 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:36:15.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:15.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:15.036 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:15.079 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:15.079 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:15.371 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.213.5"},{"key":1,"val":"00000000004.67.1"},{"key":2,"val":"00000000006.46.5"},{"key":3,"val":"00000000004.70.1"},{"key":4,"val":"00000000006.91.5"},{"key":5,"val":"00000000004.39.1"},{"key":6,"val":"00000000004.44.1"},{"key":7,"val":"00000000006.56.5"},{"key":8,"val":"00000000004.54.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.53.1"}]} 2021-02-08T07:36:15.371 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:36:15.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:15.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:15.372 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:15.460 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:15.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:15.740 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.213.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.46.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.91.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.56.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}}] 2021-02-08T07:36:15.740 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:36:15.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:15.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:15.741 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:15.781 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:15.782 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:16.057 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.213.5"},{"key":1,"val":"00000000004.67.1"},{"key":2,"val":"00000000006.46.5"},{"key":3,"val":"00000000004.70.1"},{"key":4,"val":"00000000006.91.5"},{"key":5,"val":"00000000004.39.1"},{"key":6,"val":"00000000004.44.1"},{"key":7,"val":"00000000006.56.5"},{"key":8,"val":"00000000004.54.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.53.1"}]} 2021-02-08T07:36:16.057 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:36:16.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:16.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:16.058 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:16.102 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:16.102 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:16.375 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.213.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.56.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}}] 2021-02-08T07:36:16.376 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-60 2021-02-08T07:36:16.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:16.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:16.376 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:16.418 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:16.651 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000002.264.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.81.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:16.651 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-60 2021-02-08T07:36:16.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:16.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:16.652 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:16.697 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:16.697 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.264.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.81.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.55.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:16.971 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-60 2021-02-08T07:36:16.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:16.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:16.972 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:17.011 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:17.011 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:17.239 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000002.264.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.81.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:17.239 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-60 2021-02-08T07:36:17.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:17.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:17.239 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:17.286 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:17.286 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:17.580 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.264.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.81.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.55.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:17.581 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-60 2021-02-08T07:36:17.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:17.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:17.581 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:17.624 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:17.624 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:17.897 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000002.224.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.52.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.73.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.59.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:17.898 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-60 2021-02-08T07:36:17.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:17.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:17.899 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:17.988 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:17.989 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:18.289 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.224.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.52.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.73.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:18.290 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-60 2021-02-08T07:36:18.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:18.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:18.290 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:18.333 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:18.605 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000002.224.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.52.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.73.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.59.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:18.605 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-60 2021-02-08T07:36:18.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:18.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:18.606 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:18.656 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:18.656 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:18.896 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.224.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.52.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.73.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:18.896 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-60 2021-02-08T07:36:18.897 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-60 zones={test-zone2, test-zone1} 2021-02-08T07:36:20.019 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:20.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:20.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:20.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:20.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:20.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:20.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:20.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:20.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:20.029 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:20.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:20.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:20.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:20.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:20.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:20.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:20.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:20.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:20.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:20.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:20.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:20.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:20.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:20.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:20.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:20.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:20.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:20.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:20.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:21.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:21.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:21.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:21.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:21.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:21.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:21.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:21.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:21.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:21.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:22.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:22.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:22.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:22.371 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-60 zones={test-zone2, test-zone1} 2021-02-08T07:36:22.371 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-60 zones={test-zone3, test-zone1} 2021-02-08T07:36:23.080 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:23.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:23.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:23.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:23.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:23.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:23.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:23.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:23.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:23.087 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:23.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:23.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:23.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:23.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:23.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:23.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:23.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:23.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:23.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:23.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:23.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:23.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:23.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:23.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:23.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:23.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:23.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:24.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:24.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:24.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:24.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:24.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:24.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:24.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:24.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:24.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:24.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:24.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:25.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:25.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:25.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:25.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:25.391 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-60 zones={test-zone3, test-zone1} 2021-02-08T07:36:25.392 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-60 zones={test-zone3, test-zone2} 2021-02-08T07:36:25.737 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:25.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:25.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:25.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:25.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:25.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:25.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:25.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:25.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:25.750 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:25.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:25.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:25.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:25.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:25.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:25.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:25.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:25.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:25.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:25.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:25.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:26.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:26.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:26.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:26.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:26.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:26.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:26.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:26.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:26.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:26.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:26.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:27.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:27.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:27.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:27.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:27.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:27.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:27.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:27.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:27.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:27.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:27.997 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-60 zones={test-zone3, test-zone2} 2021-02-08T07:36:27.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:27.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:27.998 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:28.038 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:28.039 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:28.312 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.74.5"},{"key":1,"val":"00000000004.70.1"},{"key":2,"val":"00000000006.73.5"},{"key":3,"val":"00000000004.44.1"},{"key":4,"val":"00000000006.46.5"},{"key":5,"val":"00000000004.45.1"},{"key":6,"val":"00000000004.74.1"},{"key":7,"val":"00000000006.215.5"},{"key":8,"val":"00000000004.48.1"},{"key":9,"val":"00000000004.45.1"},{"key":10,"val":"00000000004.59.1"}]} 2021-02-08T07:36:28.312 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:36:28.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:28.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:28.313 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:28.356 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:28.357 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:28.612 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.74.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.73.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.215.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1"}}}] 2021-02-08T07:36:28.613 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:36:28.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:28.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:28.613 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:28.655 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:28.656 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:28.915 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.74.5"},{"key":1,"val":"00000000004.70.1"},{"key":2,"val":"00000000006.73.5"},{"key":3,"val":"00000000004.44.1"},{"key":4,"val":"00000000006.46.5"},{"key":5,"val":"00000000004.45.1"},{"key":6,"val":"00000000004.74.1"},{"key":7,"val":"00000000006.215.5"},{"key":8,"val":"00000000004.48.1"},{"key":9,"val":"00000000004.45.1"},{"key":10,"val":"00000000004.59.1"}]} 2021-02-08T07:36:28.916 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:36:28.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:28.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:28.916 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:29.002 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:29.002 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:29.291 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.74.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.73.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.215.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1"}}}] 2021-02-08T07:36:29.292 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:36:29.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:29.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:29.292 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:29.378 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:29.379 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:29.646 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.74.5"},{"key":1,"val":"00000000004.70.1"},{"key":2,"val":"00000000006.73.5"},{"key":3,"val":"00000000004.44.1"},{"key":4,"val":"00000000006.46.5"},{"key":5,"val":"00000000004.45.1"},{"key":6,"val":"00000000004.74.1"},{"key":7,"val":"00000000006.215.5"},{"key":8,"val":"00000000004.48.1"},{"key":9,"val":"00000000004.45.1"},{"key":10,"val":"00000000004.59.1"}]} 2021-02-08T07:36:29.646 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:36:29.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:29.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:29.647 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:29.690 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:29.691 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:29.967 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.74.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.215.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1"}}}] 2021-02-08T07:36:29.968 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-61 2021-02-08T07:36:29.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:29.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:29.969 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:30.048 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:30.049 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:30.299 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000002.73.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.83.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.266.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:30.299 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-61 2021-02-08T07:36:30.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:30.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:30.300 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:30.344 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:30.345 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:30.642 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.73.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.83.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.266.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:30.643 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-61 2021-02-08T07:36:30.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:30.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:30.643 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:30.676 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:30.677 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:30.909 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000002.73.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.83.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.266.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:30.909 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-61 2021-02-08T07:36:30.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:30.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:30.910 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:30.955 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:30.955 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:31.246 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.73.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.83.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.266.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:31.246 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-61 2021-02-08T07:36:31.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:31.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:31.247 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:31.329 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:31.329 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:31.610 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000002.90.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.65.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.226.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:31.610 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-61 2021-02-08T07:36:31.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:31.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:31.611 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:31.693 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:31.694 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:32.001 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.90.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.65.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.226.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:32.001 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-61 2021-02-08T07:36:32.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:32.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:32.002 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:32.085 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:32.085 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:32.355 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000002.90.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.65.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.226.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:32.356 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-61 2021-02-08T07:36:32.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:32.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:32.356 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:32.648 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.90.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.65.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.226.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:32.649 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-61 2021-02-08T07:36:32.649 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-61 zones={test-zone2, test-zone1} 2021-02-08T07:36:32.754 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:32.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:32.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:32.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:32.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:32.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:32.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:32.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:32.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:32.762 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:32.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:32.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:32.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:32.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:32.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:32.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:32.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:32.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:32.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:32.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:32.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:33.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:33.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:33.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:33.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:33.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:33.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:33.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:33.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:33.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:33.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:33.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:34.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:34.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:34.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:34.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:34.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:34.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:34.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:34.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:34.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:34.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:35.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-61 zones={test-zone2, test-zone1} 2021-02-08T07:36:35.056 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-61 zones={test-zone3, test-zone1} 2021-02-08T07:36:35.367 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:35.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:35.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:35.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:35.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:35.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:35.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:35.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:35.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:35.374 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:35.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:35.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:35.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:35.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:35.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:35.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:35.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:35.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:35.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:35.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:35.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:35.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:35.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:35.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:35.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:36.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:36.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:36.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:36.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:36.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:36.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:36.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:36.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:36.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:36.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:37.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:37.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:37.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:37.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:37.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:37.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:37.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:37.688 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-61 zones={test-zone3, test-zone1} 2021-02-08T07:36:37.688 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-61 zones={test-zone3, test-zone2} 2021-02-08T07:36:38.297 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:38.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:38.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:38.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:38.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:38.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:38.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:38.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:38.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:38.312 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:38.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:38.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:38.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:38.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:38.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-08T07:36:38.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-08T07:36:38.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-08T07:36:38.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-08T07:36:38.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:38.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:38.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:38.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:38.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:38.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:39.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:39.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:39.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:39.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:39.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:39.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:39.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:39.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:39.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-08T07:36:40.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:40.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:40.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-08T07:36:40.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:40.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:40.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-08T07:36:40.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:40.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:40.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-08T07:36:40.993 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-61 zones={test-zone3, test-zone2} 2021-02-08T07:36:40.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:40.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:40.994 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:41.034 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:41.308 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769744.633820_332.1","last_update":"2021-02-08T07:35:44.633820Z"},{"marker":"1_1612769745.646227_701.1","last_update":"2021-02-08T07:35:45.646227Z"},{"marker":"1_1612769744.557430_561.1","last_update":"2021-02-08T07:35:44.557430Z"},{"marker":"1_1612769742.898104_473.3","last_update":"2021-02-08T07:35:42.898104Z"}] 2021-02-08T07:36:41.309 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-08T07:36:41.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:41.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:41.309 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:41.353 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:41.353 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:41.594 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17988498972253181383},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769744.633820_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.633820Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769745.646227_701.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:45.646227Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769744.557430_561.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.557430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769742.898104_473.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:42.898104Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:36:41.594 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-08T07:36:41.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:41.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:41.595 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:41.636 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:41.636 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:41.900 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769744.633820_332.1","last_update":"2021-02-08T07:35:44.633820Z"},{"marker":"1_1612769745.646227_701.1","last_update":"2021-02-08T07:35:45.646227Z"},{"marker":"1_1612769744.557430_561.1","last_update":"2021-02-08T07:35:44.557430Z"},{"marker":"1_1612769742.898104_473.3","last_update":"2021-02-08T07:35:42.898104Z"}] 2021-02-08T07:36:41.901 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-08T07:36:41.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:41.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:41.901 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:41.983 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:42.246 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7430472880135713257},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769744.633820_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.633820Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769745.646227_701.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:45.646227Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769744.557430_561.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.557430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769742.898104_473.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:42.898104Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:36:42.246 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-08T07:36:42.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:42.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:42.247 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:42.327 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:42.327 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:42.603 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769744.633820_332.1","last_update":"2021-02-08T07:35:44.633820Z"},{"marker":"1_1612769745.646227_701.1","last_update":"2021-02-08T07:35:45.646227Z"},{"marker":"1_1612769744.557430_561.1","last_update":"2021-02-08T07:35:44.557430Z"},{"marker":"1_1612769742.898104_473.3","last_update":"2021-02-08T07:35:42.898104Z"}] 2021-02-08T07:36:42.603 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-08T07:36:42.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:42.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:42.604 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:42.647 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:42.647 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:42.918 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":729464676215090460},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769744.633820_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.633820Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769745.646227_701.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:45.646227Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769744.557430_561.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.557430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769742.898104_473.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:42.898104Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:36:42.918 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-08T07:36:42.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:42.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:42.919 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:42.998 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:42.999 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:43.247 INFO:teuthology.orchestra.run.smithi110.stdout:[{"marker":"1_1612769751.970612_237.3","last_update":"2021-02-08T07:35:51.970612Z"},{"marker":"1_1612769751.970612_657.2","last_update":"2021-02-08T07:35:51.970612Z"},{"marker":"1_1612769751.970612_555.3","last_update":"2021-02-08T07:35:51.970612Z"},{"marker":"1_1612769751.970612_439.3","last_update":"2021-02-08T07:35:51.970612Z"}] 2021-02-08T07:36:43.247 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-08T07:36:43.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:43.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:43.248 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:43.291 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:43.571 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14959830998519150438},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769751.970612_237.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:51.970612Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769751.970612_657.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:51.970612Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769751.970612_555.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:51.970612Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769751.970612_439.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:51.970612Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:36:43.571 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-08T07:36:43.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:43.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:43.572 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:43.612 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:43.612 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:43.859 INFO:teuthology.orchestra.run.smithi110.stdout:[{"marker":"1_1612769751.970612_237.3","last_update":"2021-02-08T07:35:51.970612Z"},{"marker":"1_1612769751.970612_657.2","last_update":"2021-02-08T07:35:51.970612Z"},{"marker":"1_1612769751.970612_555.3","last_update":"2021-02-08T07:35:51.970612Z"},{"marker":"1_1612769751.970612_439.3","last_update":"2021-02-08T07:35:51.970612Z"}] 2021-02-08T07:36:43.860 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-08T07:36:43.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:43.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:43.860 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:43.904 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:43.904 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:44.172 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":711522086589135958},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769751.970612_237.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:51.970612Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769751.970612_657.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:51.970612Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769751.970612_555.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:51.970612Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769751.970612_439.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:51.970612Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:36:44.172 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-08T07:36:44.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:44.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:44.173 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:44.210 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:44.211 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:44.484 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769744.777933_210.1","last_update":"2021-02-08T07:35:44.777933Z"},{"marker":"1_1612769745.096196_504.1","last_update":"2021-02-08T07:35:45.096196Z"},{"marker":"1_1612769744.311235_440.1","last_update":"2021-02-08T07:35:44.311235Z"},{"marker":"1_1612769639.722373_279.1","last_update":"2021-02-08T07:33:59.722373Z"}] 2021-02-08T07:36:44.484 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-08T07:36:44.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:44.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:44.485 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:44.574 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:44.574 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:44.841 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12652603110286941624},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769744.777933_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.777933Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769745.096196_504.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:45.096196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769744.311235_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.311235Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769639.722373_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.722373Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:36:44.841 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-08T07:36:44.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:44.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:44.841 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:44.923 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:44.923 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:45.191 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1612769744.777933_210.1","last_update":"2021-02-08T07:35:44.777933Z"},{"marker":"1_1612769745.096196_504.1","last_update":"2021-02-08T07:35:45.096196Z"},{"marker":"1_1612769744.311235_440.1","last_update":"2021-02-08T07:35:44.311235Z"},{"marker":"1_1612769639.722373_279.1","last_update":"2021-02-08T07:33:59.722373Z"}] 2021-02-08T07:36:45.191 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-08T07:36:45.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:45.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:45.192 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:45.234 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:45.234 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:45.477 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5378454525578809946},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1612769744.777933_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.777933Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1612769745.096196_504.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:45.096196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1612769744.311235_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:35:44.311235Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1612769639.722373_279.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:33:59.722373Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-08T07:36:45.478 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-08T07:36:45.478 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2021-02-08T07:36:45.479 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lhddgo-62 2021-02-08T07:36:45.878 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lhddgo-63 2021-02-08T07:36:45.968 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lhddgo-64 2021-02-08T07:36:47.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:47.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:47.335 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:47.381 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:47.381 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:47.601 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=38375dfb-4fb1-40a7-81e1-7d3f26b8d208 2021-02-08T07:36:47.642 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612769806.702369_494.1","last_update":"2021-02-08T07:36:46.702369Z"}] 2021-02-08T07:36:47.643 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:36:47.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:47.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:47.643 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:47.682 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:47.682 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:47.909 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769806.702369_494.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:36:46.702369Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:36:47.909 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:36:47.910 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:36:47.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:47.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:47.910 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:47.952 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:47.952 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:48.190 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769806.702369_494.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:36:46.702369Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:36:48.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:36:48.190 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:36:48.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:48.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:48.191 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:48.503 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769806.702369_494.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:36:46.702369Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:36:48.504 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:36:48.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:48.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:48.505 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:48.551 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:48.551 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:48.794 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.79.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:48.794 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:36:48.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:48.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:48.794 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:48.853 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:48.853 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:49.067 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:49.067 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:36:49.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:49.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:49.068 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:49.109 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:49.110 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:49.357 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.79.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:49.357 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:36:49.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:49.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:49.358 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:49.440 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:49.441 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:49.709 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:49.710 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:36:49.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:49.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:49.710 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:49.791 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:49.792 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:50.027 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.79.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:50.027 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:36:50.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:50.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:50.027 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:50.088 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:50.089 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:50.347 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.48.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.79.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.52.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.83.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:50.347 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:36:50.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:50.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:50.348 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:50.390 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:50.391 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:50.635 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.82.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.81.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:50.636 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-62 2021-02-08T07:36:50.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:50.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:50.636 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:50.676 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:50.677 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:50.948 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:50.949 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-62 2021-02-08T07:36:50.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:50.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:50.949 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:50.992 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:50.992 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:51.231 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.82.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.81.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:51.231 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-62 2021-02-08T07:36:51.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:51.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:51.232 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:51.276 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:51.276 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:51.574 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:51.574 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-62 2021-02-08T07:36:51.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:51.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:51.575 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:51.614 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:51.614 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:51.884 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.97.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.112.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:51.885 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-62 2021-02-08T07:36:51.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:51.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:51.885 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:51.968 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:51.968 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:52.267 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.97.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.112.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:52.268 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-62 2021-02-08T07:36:52.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:52.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:52.268 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:52.351 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:52.351 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:52.630 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.97.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.112.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:52.630 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-62 2021-02-08T07:36:52.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:52.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:52.631 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:52.678 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:52.679 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:52.961 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.97.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.112.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:52.961 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-62 2021-02-08T07:36:52.962 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-62 zones={test-zone2, test-zone1} 2021-02-08T07:36:54.063 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:54.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:54.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:54.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:54.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:54.071 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:54.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:54.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:54.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:54.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:54.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:54.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:54.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:54.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:54.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:54.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:54.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:54.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:54.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:54.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:54.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:55.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:55.225 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-62 zones={test-zone2, test-zone1} 2021-02-08T07:36:55.225 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-62 zones={test-zone3, test-zone1} 2021-02-08T07:36:56.084 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:56.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:56.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:56.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:56.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:56.091 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:56.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:56.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:56.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:56.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:56.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:56.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:56.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:56.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:56.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:56.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:56.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:56.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:56.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:56.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:56.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:57.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:57.223 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-62 zones={test-zone3, test-zone1} 2021-02-08T07:36:57.224 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-62 zones={test-zone3, test-zone2} 2021-02-08T07:36:57.345 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:36:57.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:57.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:57.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:57.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:57.356 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:36:57.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:36:57.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:36:57.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:36:57.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:36:57.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:57.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:57.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:36:57.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:57.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:57.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:36:57.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:58.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:58.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:36:58.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:58.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:58.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:36:58.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-62 zones={test-zone3, test-zone2} 2021-02-08T07:36:58.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:58.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:58.507 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:58.546 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:58.547 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:58.836 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.74.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:58.836 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-63 2021-02-08T07:36:58.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:58.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:58.837 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:58.881 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:58.881 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:59.150 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:59.150 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-63 2021-02-08T07:36:59.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:59.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:59.151 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:59.193 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:59.193 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:59.486 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.74.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:36:59.487 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-63 2021-02-08T07:36:59.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:59.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:59.487 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:59.577 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:59.577 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:36:59.853 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:36:59.853 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-63 2021-02-08T07:36:59.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:59.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:36:59.854 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:36:59.892 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:36:59.892 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:00.179 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.74.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:00.180 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-63 2021-02-08T07:37:00.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:00.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:00.181 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:00.225 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:00.225 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:00.500 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.57.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.74.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:00.500 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-63 2021-02-08T07:37:00.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:00.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:00.501 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:00.581 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:00.581 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:00.836 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.63.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:00.836 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-63 2021-02-08T07:37:00.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:00.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:00.837 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:00.881 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:00.881 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:01.188 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:01.189 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-63 2021-02-08T07:37:01.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:01.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:01.190 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:01.231 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:01.231 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:01.487 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.63.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:01.487 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-63 2021-02-08T07:37:01.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:01.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:01.488 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:01.533 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:01.533 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:01.827 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:01.827 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-63 2021-02-08T07:37:01.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:01.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:01.828 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:01.912 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:01.913 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:02.179 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.51.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.71.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.71.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:02.179 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-63 2021-02-08T07:37:02.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:02.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:02.180 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:02.269 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:02.270 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:02.574 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:02.575 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-63 2021-02-08T07:37:02.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:02.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:02.575 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:02.616 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:02.616 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:02.881 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.51.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.71.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.71.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:02.881 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-63 2021-02-08T07:37:02.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:02.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:02.882 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:02.918 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:02.918 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:03.187 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:03.188 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-63 2021-02-08T07:37:03.188 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-63 zones={test-zone2, test-zone1} 2021-02-08T07:37:03.903 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:37:03.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:03.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:03.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:03.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:03.910 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:37:03.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:03.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:03.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:03.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:03.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:03.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:04.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:04.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:04.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:04.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:04.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:04.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:04.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:04.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:04.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:05.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:05.163 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-63 zones={test-zone2, test-zone1} 2021-02-08T07:37:05.163 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-63 zones={test-zone3, test-zone1} 2021-02-08T07:37:06.075 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:37:06.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:06.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:06.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:06.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:06.080 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:37:06.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:06.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:06.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:06.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:06.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:06.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:06.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:06.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:06.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:06.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:06.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:06.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:06.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:06.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:07.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:07.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:07.252 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-63 zones={test-zone3, test-zone1} 2021-02-08T07:37:07.253 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-63 zones={test-zone3, test-zone2} 2021-02-08T07:37:07.377 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:37:07.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:07.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:07.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:07.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:07.385 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:37:07.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:07.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:07.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:07.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:07.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:07.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:07.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:07.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:07.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:07.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:07.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:08.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:08.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:08.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:08.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:08.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:08.485 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-63 zones={test-zone3, test-zone2} 2021-02-08T07:37:08.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:08.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:08.486 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:08.526 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:08.527 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:08.794 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.95.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.59.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:08.795 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-64 2021-02-08T07:37:08.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:08.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:08.795 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:08.840 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:08.841 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:09.114 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:09.114 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-64 2021-02-08T07:37:09.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:09.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:09.115 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:09.157 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:09.158 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:09.431 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.95.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.59.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:09.431 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-64 2021-02-08T07:37:09.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:09.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:09.432 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:09.519 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:09.519 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:09.802 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:09.803 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-64 2021-02-08T07:37:09.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:09.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:09.803 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:09.883 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:09.883 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:10.163 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.95.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.59.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:10.163 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-64 2021-02-08T07:37:10.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:10.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:10.164 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:10.209 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:10.209 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:10.484 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.61.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.95.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.59.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:10.485 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-64 2021-02-08T07:37:10.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:10.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:10.485 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:10.574 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:10.574 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:10.822 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.65.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.85.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.58.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:10.822 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-64 2021-02-08T07:37:10.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:10.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:10.823 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:10.871 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:10.871 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:11.175 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:11.176 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lhddgo-64 2021-02-08T07:37:11.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:11.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:11.176 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:11.471 INFO:teuthology.orchestra.run.smithi110.stdout:{"markers":[{"key":0,"val":"00000000001.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.65.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.85.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.58.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:11.472 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-64 2021-02-08T07:37:11.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:11.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:11.472 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lhddgo-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:11.518 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:11.519 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:11.829 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:11.829 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lhddgo-64 2021-02-08T07:37:11.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:11.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:11.830 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:11.919 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:11.919 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:12.176 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.69.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.67.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.79.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.63.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:12.176 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-64 2021-02-08T07:37:12.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:12.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:12.177 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:12.261 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:12.558 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:12.559 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lhddgo-64 2021-02-08T07:37:12.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:12.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:12.559 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:12.638 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:12.638 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:12.922 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.69.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.67.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.79.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.63.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-02-08T07:37:12.923 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-64 2021-02-08T07:37:12.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:12.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lhddgo-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:12.923 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lhddgo-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:12.968 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:12.968 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:14.052 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:14.053 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lhddgo-64 2021-02-08T07:37:14.053 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-64 zones={test-zone2, test-zone1} 2021-02-08T07:37:14.922 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:37:14.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:14.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:14.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:14.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:14.930 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:37:14.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:14.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:14.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:14.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:14.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:15.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:15.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:15.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:15.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:15.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:15.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:15.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:15.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:15.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:15.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:15.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:16.103 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-64 zones={test-zone2, test-zone1} 2021-02-08T07:37:16.103 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-64 zones={test-zone3, test-zone1} 2021-02-08T07:37:16.555 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:37:16.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:16.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:16.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:16.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:16.561 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:37:16.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:16.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:16.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:16.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:16.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:16.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:16.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:16.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:16.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:17.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:17.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:17.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:17.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:17.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:17.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:17.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:17.720 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-64 zones={test-zone3, test-zone1} 2021-02-08T07:37:17.720 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lhddgo-64 zones={test-zone3, test-zone2} 2021-02-08T07:37:18.505 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-08T07:37:18.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:18.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:18.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:18.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:18.515 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-08T07:37:18.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-08T07:37:18.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-08T07:37:18.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-08T07:37:18.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-08T07:37:18.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:18.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:18.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-08T07:37:18.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:18.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:18.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-08T07:37:19.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:19.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:19.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-08T07:37:19.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:19.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:19.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-08T07:37:19.652 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lhddgo-64 zones={test-zone3, test-zone2} 2021-02-08T07:37:19.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'lhddgo-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:19.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'lhddgo-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:19.653 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket lhddgo-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:37:19.696 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:19.696 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:20.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'lhddgo-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:20.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'lhddgo-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:20.036 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket lhddgo-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:37:20.119 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:20.120 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:20.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'lhddgo-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:20.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'lhddgo-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:20.463 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket lhddgo-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:37:20.548 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:20.548 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:20.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:20.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:20.867 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:20.943 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:20.943 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=38375dfb-4fb1-40a7-81e1-7d3f26b8d208 2021-02-08T07:37:21.220 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1612769840.823975_519.1","last_update":"2021-02-08T07:37:20.823975Z"}] 2021-02-08T07:37:21.220 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-08T07:37:21.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:21.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:21.221 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:21.265 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:21.265 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:21.476 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769840.823975_519.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:37:20.823975Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:37:21.476 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-08T07:37:21.477 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-08T07:37:21.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:21.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:21.477 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:21.565 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:21.565 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:21.816 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769839.986940_513.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:37:19.986940Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:37:21.816 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1612769840.823975_519.1 target=1_1612769839.986940_513.1 2021-02-08T07:37:26.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:26.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:26.818 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:26.858 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:26.858 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:27.129 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769840.823975_519.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:37:20.823975Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:37:27.129 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-08T07:37:27.129 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-08T07:37:27.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:27.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:27.130 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:27.450 INFO:teuthology.orchestra.run.smithi110.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"38375dfb-4fb1-40a7-81e1-7d3f26b8d208","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1612768579.532632_32.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-08T07:16:19.532632Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1612768739.752484_117.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:18:59.752484Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1612769840.823975_519.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-08T07:37:20.823975Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-02-08T07:37:27.511 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-08T07:37:29.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'lhddgo-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:29.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'lhddgo-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:29.407 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket lhddgo-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:37:29.449 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:29.449 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:29.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'lhddgo-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:29.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'lhddgo-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:29.784 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket lhddgo-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:37:29.878 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:29.878 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:30.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'lhddgo-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:30.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'lhddgo-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-08T07:37:30.191 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket lhddgo-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-08T07:37:30.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:30.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:30.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:30.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:30.577 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:30.660 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:30.661 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:30.959 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.58.1"},{"key":1,"val":"00000000004.50.1"},{"key":2,"val":"00000000006.89.1"},{"key":3,"val":"00000000004.80.1"},{"key":4,"val":"00000000006.62.1"},{"key":5,"val":"00000000004.51.1"},{"key":6,"val":"00000000004.78.1"},{"key":7,"val":"00000000006.91.1"},{"key":8,"val":"00000000004.39.1"},{"key":9,"val":"00000000004.71.1"},{"key":10,"val":"00000000004.60.1"}]} 2021-02-08T07:37:30.959 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:37:30.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:30.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:30.960 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:31.009 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:31.263 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:31.264 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone2 behind zone test-zone1: shard=0 master=00000000006.58.1 target=, shard=1 master=00000000004.50.1 target=, shard=2 master=00000000006.89.1 target=, shard=4 master=00000000006.62.1 target=, shard=5 master=00000000004.51.1 target=, shard=6 master=00000000004.78.1 target=, shard=8 master=00000000004.39.1 target=, shard=9 master=00000000004.71.1 target=, shard=10 master=00000000004.60.1 target= 2021-02-08T07:37:36.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:36.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:36.265 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:36.309 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:36.309 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:36.567 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-02-08T07:37:36.568 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone2 behind zone test-zone1: shard=0 master=00000000006.58.1 target=, shard=1 master=00000000004.50.1 target=, shard=2 master=00000000006.89.1 target=, shard=4 master=00000000006.62.1 target=, shard=5 master=00000000004.51.1 target=, shard=6 master=00000000004.78.1 target=, shard=8 master=00000000004.39.1 target=, shard=9 master=00000000004.71.1 target=, shard=10 master=00000000004.60.1 target= 2021-02-08T07:37:41.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:41.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:41.569 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:41.613 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:41.613 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:41.830 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.58.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.89.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.62.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:37:41.830 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:37:41.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:41.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:41.831 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:41.877 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:41.878 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:42.153 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.58.1"},{"key":1,"val":"00000000004.50.1"},{"key":2,"val":"00000000006.89.1"},{"key":3,"val":"00000000004.80.1"},{"key":4,"val":"00000000006.62.1"},{"key":5,"val":"00000000004.51.1"},{"key":6,"val":"00000000004.78.1"},{"key":7,"val":"00000000006.91.1"},{"key":8,"val":"00000000004.39.1"},{"key":9,"val":"00000000004.71.1"},{"key":10,"val":"00000000004.60.1"}]} 2021-02-08T07:37:42.153 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:37:42.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:42.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:42.154 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:42.197 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:42.197 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:42.546 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.58.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.89.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.62.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:37:42.547 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:37:42.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:42.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lhddgo-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:42.547 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lhddgo-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:42.626 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:42.626 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:42.881 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.58.1"},{"key":1,"val":"00000000004.50.1"},{"key":2,"val":"00000000006.89.1"},{"key":3,"val":"00000000004.80.1"},{"key":4,"val":"00000000006.62.1"},{"key":5,"val":"00000000004.51.1"},{"key":6,"val":"00000000004.78.1"},{"key":7,"val":"00000000006.91.1"},{"key":8,"val":"00000000004.39.1"},{"key":9,"val":"00000000004.71.1"},{"key":10,"val":"00000000004.60.1"}]} 2021-02-08T07:37:42.882 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lhddgo-62 2021-02-08T07:37:42.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:42.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:42.882 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:42.925 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:42.925 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:43.207 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:37:43.208 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:37:48.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:48.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:48.209 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:48.532 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:37:48.533 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:37:53.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:53.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:53.534 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:53.577 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:53.578 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:53.840 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:37:53.840 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:37:58.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:58.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:37:58.842 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:37:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:37:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:37:59.161 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:37:59.162 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:04.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:04.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:04.163 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:04.206 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:04.206 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:04.486 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:04.487 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:09.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:09.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:09.489 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:09.533 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:09.533 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:09.807 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:09.807 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:14.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:14.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:14.809 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:14.853 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:14.853 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:15.126 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:15.126 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:20.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:20.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:20.127 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:20.170 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:20.170 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:20.446 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:20.446 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:25.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:25.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:25.449 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:25.494 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:25.495 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:25.773 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:25.774 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:30.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:30.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:30.775 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:30.821 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:30.821 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:31.099 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:31.099 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:36.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:36.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:36.101 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:36.150 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:36.151 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:36.427 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:36.427 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:41.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:41.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:41.429 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:41.474 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:41.474 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:41.764 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:41.765 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:46.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:46.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:46.766 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:46.811 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:46.811 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:47.091 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:47.091 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:52.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:52.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:52.092 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:52.141 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:52.142 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:52.425 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:52.425 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:38:57.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:57.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:38:57.426 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:38:57.469 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:38:57.469 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:38:57.752 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:38:57.753 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:39:02.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:39:02.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:39:02.754 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:39:02.797 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:39:02.798 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T07:39:03.072 INFO:teuthology.orchestra.run.smithi110.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.78.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}}] 2021-02-08T07:39:03.073 WARNING:rgw_multi.tests:bucket lhddgo-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.58.1 target=00000000002.49.1, shard=2 master=00000000006.89.1 target=00000000002.80.1, shard=4 master=00000000006.62.1 target=00000000002.55.1 2021-02-08T07:39:08.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:39:08.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lhddgo-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-08T07:39:08.074 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lhddgo-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-08T07:39:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2021-02-08T07:39:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2021-02-08T19:05:28.816 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-02-08T19:05:29.344 DEBUG:teuthology.task.console_log:Killing console logger for smithi017 2021-02-08T19:05:29.345 DEBUG:teuthology.task.console_log:Killing console logger for smithi110 2021-02-08T19:05:29.346 DEBUG:teuthology.exit:Finished running handlers